[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds. http://wiki.apache.org/couchdb/Performance?action=diff&rev1=3&rev2=4 -- As an example one of my benchmarks turned out to be mostly limited by the json module's encoding and decoding speed. The process was using 40% of a CPU. Switching to simplejson with no other changes resulted in 5% of a CPU. Switching from threads to processes (using multiprocessing module) gave yet another performance improvement finally pushing CouchDB to consume more than 100% of a CPU (this is on a multi-processor machine). + = Resource Limits = + One of the problems that administrators run into as their deployments become large are resource limits imposed by the system and by the application configuration. Raising these limits can allow your deployment to grow beyond what the default configuration will support. + == CouchDB Configuration Options == + In your configuration (local.ini or similar) familiarize yourself with the following options:{{{ + [couchdb] + max_dbs_open = 100 + + [httpd] + max_connections = 2048}}} + The first option places an upper bound on the number of databases that can be open at one time. CouchDB reference counts database accesses internally and will close idle databases when it must. Sometimes it is necessary to keep more than the default open at once, such as in deployments where many databases will be continuously replicating. + The second option limits how many client connections the HTTP server will service at a time. Again, heavy replication scenarios are good candidates for increased {{{max_connections}}} since the replicator opens several connections to the source database. + == System Resource Limits == + === Erlang === + Even if you've increased the maximum connections CouchDB will allow, the Erlang runtime system will not allow more than 1024 connections by default. Setting the following option in {{{(prefix)/etc/default/couchdb}}} (or equivalent) will increase this limit (in this case to 4096): {{{export ERL_MAX_PORTS=4096}}} + === PAM and ulimit === + Finally, most *nix operating systems impose various resource limits on every process. If your system is set up to use the Pluggable Authentication Modules (PAM) system, increasing this limit is straightforward. For example, creating a file named {{{/etc/security/limits.d/100-couchdb.conf}}} with the following contents will ensure that CouchDB can open enough file descriptors to service your increased maximum open databases and Erlang ports:{{{ + # + couchdb hard nofile4096 + couchdb soft nofile4096}}} + If your system does not use PAM, a {{{ulimit}}} command is usually available for use in a custom script to launch CouchDB with increased resource limits. + If necessary, feel free to increase this limits as long as your hardware can handle the load. +
[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds. http://wiki.apache.org/couchdb/Performance?action=diff&rev1=4&rev2=5 -- The second option limits how many client connections the HTTP server will service at a time. Again, heavy replication scenarios are good candidates for increased {{{max_connections}}} since the replicator opens several connections to the source database. == System Resource Limits == === Erlang === - Even if you've increased the maximum connections CouchDB will allow, the Erlang runtime system will not allow more than 1024 connections by default. Setting the following option in {{{(prefix)/etc/default/couchdb}}} (or equivalent) will increase this limit (in this case to 4096): {{{export ERL_MAX_PORTS=4096}}} + Even if you've increased the maximum connections CouchDB will allow, the Erlang runtime system will not allow more than 1024 connections by default. Adding the following directive to {{{(prefix)/etc/default/couchdb}}} (or equivalent) will increase this limit (in this case to 4096):{{{ + export ERL_MAX_PORTS=4096}}} === PAM and ulimit === Finally, most *nix operating systems impose various resource limits on every process. If your system is set up to use the Pluggable Authentication Modules (PAM) system, increasing this limit is straightforward. For example, creating a file named {{{/etc/security/limits.d/100-couchdb.conf}}} with the following contents will ensure that CouchDB can open enough file descriptors to service your increased maximum open databases and Erlang ports:{{{ #
[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds. The comment on this change is: Add information on +A flag. http://wiki.apache.org/couchdb/Performance?action=diff&rev1=5&rev2=6 -- If your system does not use PAM, a {{{ulimit}}} command is usually available for use in a custom script to launch CouchDB with increased resource limits. If necessary, feel free to increase this limits as long as your hardware can handle the load. + = Disk and File System Performance = + Using faster disks, striped RAID arrays and modern file systems can all speed up your CouchDB deployment. However, there is one option that can increase the responsiveness of your CouchDB server when disk performance is a bottleneck. From the erlang documentation for the file module: {{{ + On operating systems with thread support, it is possible to let file operations be performed in threads of their own, allowing other Erlang processes to continue executing in parallel with the file operations. See the command line flag +A in erl(1).}}} + Setting this argument to a number greater than zero can keep your CouchDB installation responsive even during periods of heavy disk utilization. The easiest way to set this option is through the {{{ERL_FLAGS}}} environment variable. For example, to give Erlang four threads with which to perform i/o operations add the following to {{{(prefix)/etc/defaults/couchdb}}} (or equivalent): {{{ + export ERL_FLAGS="+A 4"}}} +
[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds. The comment on this change is: Add table of contents. http://wiki.apache.org/couchdb/Performance?action=diff&rev1=6&rev2=7 -- + <> + With up to tens of thousands of documents you will generally find CouchDB to perform well no matter how you write your code. Once you start getting into the millions of documents you need to be a lot more careful. Many of the individual wiki pages mention performance when describing how to do things. It is worthwhile refreshing your memory by revisiting them.
[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds: http://wiki.apache.org/couchdb/Performance?action=diff&rev1=8&rev2=9 Comment: Add information about script stack quota }}} The first option places an upper bound on the number of databases that can be open at one time. CouchDB reference counts database accesses internally and will close idle databases when it must. Sometimes it is necessary to keep more than the default open at once, such as in deployments where many databases will be continuously replicating. The second option limits how many client connections the HTTP server will service at a time. Again, heavy replication scenarios are good candidates for increased {{{max_connections}}} since the replicator opens several connections to the source database. + == JavaScript View Server == + As of CouchDB 1.1.1+ the JavaScript view server that's included with CouchDB has an option to tune the stack size for VM contexts. This option can be used if errors in the log indicate that the stack quota has been exceeded. The value provided will be passed to [[JS_NewContext|https://developer.mozilla.org/en/JS_NewContext]]. + + {{{ + [query_servers] + javascript = /usr/local/bin/couchjs --stack-size 81920 /usr/local/share/couchdb/server/main.js + }}} + + The example above shows a sample configuration which raises the stack size to 80MB from the default of 8MB. + == System Resource Limits == === Erlang === Even if you've increased the maximum connections CouchDB will allow, the Erlang runtime system will not allow more than 1024 connections by default. Adding the following directive to {{{(prefix)/etc/default/couchdb}}} (or equivalent) will increase this limit (in this case to 4096):
[Couchdb Wiki] Update of "Performance" by RandallLeeds
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Couchdb Wiki" for change notification. The "Performance" page has been changed by RandallLeeds: http://wiki.apache.org/couchdb/Performance?action=diff&rev1=14&rev2=15 Comment: Add ERL_MAX_ETS_TABLES note. {{{ export ERL_MAX_PORTS=4096 }}} + + CouchDB versions up to 1.1.x also create Erlang Term Storage (ETS) tables for each replication. If you are using a version of CouchDB older than 1.2 and must support many replications, also set the {{{ERL_MAX_ETS_TABLES}}} variable. The default is approximately 1400 tables. === PAM and ulimit === Finally, most *nix operating systems impose various resource limits on every process. If your system is set up to use the Pluggable Authentication Modules (PAM) system, increasing this limit is straightforward. For example, creating a file named {{{/etc/security/limits.d/100-couchdb.conf}}} with the following contents will ensure that CouchDB can open enough file descriptors to service your increased maximum open databases and Erlang ports: