On 16/11/12 19:05, Dave Cottlehuber wrote:
Many thanks for all replies on my other thread!
Erlang is amazing, and CouchDB obviously, but there's a big hurdle between
the handful of erlang books, and real-world expert development.
My (selfish!) original intent of asking for this list to be created was
to use CouchDB as a convenient playground / reference point for improving
my erlangz, and ultimately end up with some contributable code. And I
hope that this approach will be a great thing for many of us in the
same situation, and in the long term, also grow our community &
committers too.
So I am *really* keen for this to be as much of a group approach as possible,
and spend time reviewing other's solutions etc, more so than just hacking
away in a closet. We could set up a shared git repo if you all like, or
just swap patches/gists over email. Whatever works.
If you like the approach, the first one I suggest is about configuration, and we
can check back in a week. I'd be rapt if this works out and others
propose some topics along the way too.
Homework :-)
1. how are configs loaded at startup in CouchDB?
2. how they're subsequently mutated/abused over the _config REST API?
3. how are these events managed elsewhere e.g. in couch httpd when IP changes?
4. compare with the sys.config approach used in OTP [1]
5. can you see a way to have the best of both worlds?
I'll send my findings through early next week.
A+
Dave
[1]: http://www.erlang.org/doc/design_principles/applications.html#id74029
Homework.
Using rcouch on a raspbian Raspberry Pi
rcouch for this exercise is roughly Couchdb 1.3
Using erlang view server code
lists:foreach(fun(X)-> Log(io_lib:format("~p~n",[ets:info(X)])) end, ets:all())
to list all ets tables, to find couch_config below
2012-11-18 21:23:03.904 [info] <0.2383.78>@couch_native_process:bindings:273
[{compressed,false},
{memory,11004},
{owner,<0.133.0>},
{heir,none},
{name,couch_config},
{size,97},
{node,'[email protected]'},
{named_table,true}, %Name can be used to access
{type,set}, %fast access
{keypos,1}, %key is tuple of two strings
(really lists)
{protection,protected}] %any process that has table identifier
can read
then
ets:foldl(fun(Z,A)->Log(io_lib:format("~p~n",[Z])) end, [],couch_config)
to list all keys (which are tuples of strings) and values (tuples of various
forms)
All output of is to couch.log (truncated for brevity)
2012-11-18 21:33:
39.835 [info] <0.2383.78>@couch_native_process:bindings:273
{{"httpd_db_handlers","_view_cleanup"},"{couch_mrview_http,
handle_cleanup_req}"}
39.839 {{"daemons","compaction_daemon"},"{couch_compaction_daemon, start_link,
[]}"}
39.846 {{"couch_httpd_auth","iterations"},"10000"}
39.854 {{"attachments","compression_level"},"8"}
39.861 {{"replicator","http_connections"},"20"}
39.868 {{"httpd_db_handlers","_spatial_cleanup"},"{refuge_spatial_http,
handle_cleanup_req}"}
39.875 {{"couchdb","delayed_commits"},"true"}
39.882 {{"httpd_db_handlers","_random_doc"},"{couch_randomdoc_httpd,
handle_req}"}
39.889 {{"httpd_global_handlers","_config"},"{couch_httpd_misc_handlers,
handle_config_req}"}
39.896 {{"log","level"},"info"}
39.903 {{"couchdb","uri_file"},"./data/couch.uri"}
39.910 {{"httpd_db_handlers","_all_docs"},"{couch_mrview_http,
handle_all_docs_req}"}
39.917 {{"httpd","log_max_chunk_size"},"1000000"}
39.926 {{"couch_httpd_auth","authentication_redirect"},"/_utils/session.html"}
39.929 {{"couchdb","max_dbs_open"},"100"}
39.932 {{"httpd_global_handlers","_replicate"},"{couch_replicator_httpd,
handle_req}"}
39.934 {{"vendor","version"},"0.6.0"}
39.938 {{"stats","samples"},"[0, 60, 300, 900]"}
39.945 {{"query_servers","javascript"},"./bin/couchjs ./share/server/main.js"}
39.952 {{"couchdb","database_dir"},"./data"}
39.959 {{"attachments","compressible_types"},"text/*, application/javascript,
application/json, application/xml"}
39.966 {{"uuids","algorithm"},"sequential"}
39.972 {{"couch_httpd_auth","authentication_db"},"_users"}
39.980 {{"httpd_global_handlers","_log"},"{couch_httpd_misc_handlers,
handle_log_req}"}
39.987 {{"replicator","connection_timeout"},"60000"}
39.994 {{"httpd_design_handlers","_spatial"},"{refuge_spatial_http,
handle_spatial_req}"}
40.001 {{"httpd","authentication_handlers"},
"{couch_httpd_oauth, oauth_authentication_handler},
{couch_httpd_auth, cookie_authentication_handler},
{couch_httpd_auth, default_authentication_handler}"}
40.008 {{"compaction_daemon","min_file_size"},"131072"}
40.015 {{"httpd_design_handlers","_list"}, "{couch_mrview_show,
handle_view_list_req}"}
40.022 {{"httpd_design_handlers","_spatial/_list"},"{refuge_spatial_show,
handle_spatial_list_req}"}
40.029 {{"daemons","external_manager"},"{couch_external_manager, start_link,
[]}"}
40.036 {{"vendor","name"},"rcouch"}
40.043 {{"httpd","allow_jsonp"},"false"}
40.050 {{"httpd_global_handlers","_oauth"},"{couch_httpd_oauth,
handle_oauth_req}"}
40.058 {{"httpd_global_handlers","_stats"},"{couch_httpd_stats_handlers,
handle_stats_req}"}
40.065 {{"httpd_global_handlers","/"},"{couch_httpd_misc_handlers, handle_welcome_req,
<<\"Welcome\">>}"}
40.072 {{"stats","rate"},"1000"}
40.079 {{"httpd_global_handlers","_uuids"},"{couch_httpd_misc_handlers,
handle_uuids_req}"}
40.086 {{"httpd_global_handlers","_utils"},"{couch_httpd_misc_handlers, handle_utils_dir_req,
\"./share/www\"}"}
40.094 {{"httpd_db_handlers","_changes"},"{couch_httpd_changes,
handle_changes_req}"}
40.101 {{"httpd_design_handlers","_view"},"{couch_mrview_http,
handle_view_req}"}
40.108 {{"query_server_config","reduce_limit"},"true"}
40.115 {{"httpd_global_handlers","favicon.ico"},"{couch_httpd_misc_handlers,
handle_favicon_req, \"./share/www\"}"}
40.122 {{"log","file"},"./log/console.log"}
40.124 {{"couch_httpd_auth","require_valid_user"},"false"}
40.127 {{"httpd","secure_rewrites"},"true"}
40.134 {{"couch_httpd_auth","secret"},"c6e697df9db3a0b9ee7c62fc6657bc9c"}
40.140 {{"admins","david"},
"-pbkdf2-1a549048f941268c8c29140c4500a0a204dc6a88,3682c7bbbe8607b10088467218b7023c,10000"}
40.147 {{"httpd_global_handlers","_db_updates"},"{couch_dbupdates_httpd,
handle_req}"}
40.154 {{"httpd_db_handlers","_compact"},"{couch_httpd_db, handle_compact_req}"}
40.162 {{"ssl","enable"},"false"}
40.169 {{"httpd_db_handlers","_temp_view"},"{couch_mrview_http,
handle_temp_view_req}"}
40.181 {{"couch_httpd_oauth","use_users_db"},"false"}
40.184 {{"query_servers","coffeescript"},"./bin/couchjs
./share/server/main-coffee.js"}
40.187 {{"couch_httpd_auth","timeout"},"600"}
40.191 {{"replicator","db"},"_replicator"}
40.199 {{"httpd_global_handlers","_restart"},"{couch_httpd_misc_handlers,
handle_restart_req}"}
40.206 {{"httpd_design_handlers","_compact"},"{couch_mrview_http,
handle_compact_req}"}
40.213 {{"log","include_sasl"},"true"}
40.224 {{"ssl","ssl_certificate_max_depth"},"1"}
40.232 {{"httpd_design_handlers","_info"},"{couch_mrview_http,
handle_info_req}"}
40.239 {{"database_compaction","doc_buffer_size"},"524288"}
40.248 {{"httpd","default_handler"},"{couch_httpd_db, handle_request}"}
40.255 {{"httpd_design_handlers","_spatial/_info"},"{refuge_spatial_http,
handle_info_req}"}
40.264 {{"httpd","port"},"5984"}
40.269 {{"httpd_global_handlers","_active_tasks"}, "{couch_httpd_misc_handlers,
handle_task_status_req}"}
40.279 {{"couchdb","max_document_size"},"4294967296"}
40.285 {{"daemons","query_servers"},"{couch_query_servers, start_link, []}"}
40.292 {{"httpd_db_handlers","_design"},"{couch_httpd_db, handle_design_req}"}
40.300 {{"couchdb","os_process_timeout"},"500000"}
40.308 {{"httpd_design_handlers","_rewrite"},"{couch_httpd_rewrite,
handle_rewrite_req}"}
40.314 {{"replicator","socket_options"},"[{keepalive, true}, {nodelay, false}]"}
40.322 {{"httpd_design_handlers","_spatial/_compact"},"{refuge_spatial_http,
handle_compact_req}"}
40.329 {{"httpd","vhost_global_handlers"},"_utils, _uuids, _session, _oauth,
_users"}
40.336 {{"daemons","os_daemons"},"{couch_os_daemons, start_link, []}"}
40.343 {{"httpd_design_handlers","_update"},"{couch_mrview_show,
handle_doc_update_req}"}
40.350 {{"httpd","bind_address"},"0.0.0.0"}
40.357 {{"ssl","verify_ssl_certificates"},"false"}
40.364 {{"httpd_design_handlers","_random"},"{couch_randomdoc_show,
handle_randomdoc_show_req}"}
40.371 {{"replicator","max_replication_retry_count"},"10"}
40.377 {{"ssl","port"},"6986"}
40.385 {{"database_compaction","checkpoint_after"},"5242880"}
40.392 {{"replicator","verify_ssl_certificates"},"false"}
40.400 {{"view_compaction","keyvalue_buffer_size"},"2097152"}
40.402 {{"replicator","retries_per_request"},"10"}
40.405 {{"httpd_design_handlers","_show"},"{couch_mrview_show,
handle_doc_show_req}"}
40.408 {{"couchdb","index_dir"},"./data"}
40.411 {{"couchdb","file_compression"},"snappy"}
40.413 {{"replicator","worker_processes"},"4"}
40.416 {{"native_query_servers","erlang"},"{couch_native_process, start_link,
[]}"}
40.425 {{"couch_httpd_auth","auth_cache_size"},"50"}
40.432 {{"httpd_global_handlers","_all_dbs"},"{couch_httpd_misc_handlers,
handle_all_dbs_req}"}
40.463 {{"replicator","worker_batch_size"},"500"}
40.481 {{"compaction_daemon","check_interval"},"300"}
40.495 {{"replicator","ssl_certificate_max_depth"},"3"}
40.517 {{"httpd_global_handlers","_session"},"{couch_httpd_auth,
handle_session_req}"}
{{"query_server_config","os_process_limit"},"25"}
The reading of default.ini and local.ini into this ets table is another
exercise, but it should be noted
that as the couch_config table is of type "set" (all keys must be unique)
any entry from default.ini will be overwritten by any entry from
local.ini or any changes in Configuration from Futon etc.
Information in couch_config is available to all processes in couchdb without
needing to be passed any
data. Thus it is the Erlang way of having a shared fast look-up table in
memory. Joe Armstrong quotes
that a set lookup took 0.5 microsecond on his machine (2007) "Programming Erlang-
Software for a concurrent world"
pp. 277
I hope this helps someone else
David Martin (davidoccam)
