Design document "versioning".
Hi,
When using couchdb views it would be very useful to be able to see when
the version of a design document in the database needs to be updated.
Right now, I've solved this in a very inefficient way: Every time I call
a view, I first query the design document it lives in, and compare the
version number (a field I've added) of the design doc, to what the
application has in its version on disk, and if the two are out of date
(or if the design document is not there at all yet) I PUT the version
the application has.
It would be very useful if version_number (or a field like it) was made
an official field, and output along with any view results, something like:
{
'design_document_version': '0.1',
'rows': [...]
}
so that one could just call the view, and compare version numbers after
the fact, and most of the time only a single call needs to be made.
Do others think this is a good idea?
--
eric casteleijn
https://code.launchpad.net/~thisfred
Canonical Ltd.
[jira] Reopened: (COUCHDB-491) Support replication over SSL
[ https://issues.apache.org/jira/browse/COUCHDB-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn reopened COUCHDB-491: - Actually this is still not working in 1.0.1 or trunk. Filipe is working on this, but it does not seem this was ever really fixed before. > Support replication over SSL > > > Key: COUCHDB-491 > URL: https://issues.apache.org/jira/browse/COUCHDB-491 > Project: CouchDB > Issue Type: Improvement > Reporter: eric casteleijn > Fix For: 0.10 > > > CouchDB currently does not support replication over SSL which is a problem > when replicating non public databases between two CouchDB servers over the > internet. > It seems SSL support is almost there, though: > If ssl is started from bin/couchdb, push replication to a couchdb that is > behind an SSL proxy works (i.e. to an https:// url), but pull replication > from same fails, because apparently the request is not encrypted at all. (At > least nginx seemed to think it wasn't.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Futon cannot use the RESTful HTTP API
On 02/19/2010 06:43 AM, Florian Weimer wrote: * Paul Davis: Do you have any examples of how other sites have protected against this? Unless I'm missing something I don't see how this is specific to Futon so surely someone else has some explicit documentation on how to avoid such things. The standard countermeasure puts session identifiers into URLs (sometimes they are called "form tokens", but this doesn't fit the context here). Upon login, the client could specify if it wants a session with or without form tokens. This: http://www.cgisecurity.com/csrf-faq.html is a pretty thorough resource. One of the first discoveries of the vulnerability was in Zope back in 2000. AFAIK, the issue has still not been fixed there (although it has been in some CMSes built on top of Zope), which tells you something about Zope, but also about how exploitable this bug is: You need to target a specific site that you know is running a particular back end, craft a POST url that will do something nasty, and then trick someone with admin privileges into clicking that link, while being logged in to the targeted site as admin. I'm not saying it shouldn't be fixed, (with nonce tokens in any form that does something potentially dangerous, as Florian says) but it's not threat level 'change underwear'. eric 'someday soon I'll write a whole mail without mentioning the good old days of Zope' casteleijn
[jira] Updated: (COUCHDB-647) Zero size DB files are created, which make CouchDB crash.
[ https://issues.apache.org/jira/browse/COUCHDB-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn updated COUCHDB-647: Attachment: couch_file_logging.patch Written by Chris Anderson, so he should be the one to grant license. > Zero size DB files are created, which make CouchDB crash. > - > > Key: COUCHDB-647 > URL: https://issues.apache.org/jira/browse/COUCHDB-647 > Project: CouchDB > Issue Type: Bug >Affects Versions: 0.9.1, 0.10 > Environment: Ubuntu Hardy > Reporter: eric casteleijn >Priority: Critical > Attachments: couch_file_logging.patch > > > Our production server crashed with the following fragment in the error log > http://friendpaste.com/3VfsxGrH2XxvkqE3XQA4oy > It appears that this is due to a corrupted or zero size database file. > Chris Anderson suggested the attached patch to improve the logging in this > case. > doppler came up with a script that reproduces the crash: > touch > /Applications/CouchDBX-0.10.1-R13B03-Leopard.app/Contents/Resources/couchdbx-core/couchdb/var/lib/couchdb/test.couch > while true ; do curl -X GET http://localhost:5984/test ; done > NOTE: On the server that crashed we do not manipulate database files in any > other way than calling the REST interface, so it is still a mystery how zero > sized dbs get created. I will investigate by digging through the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-647) Zero size DB files are created, which make CouchDB crash.
Zero size DB files are created, which make CouchDB crash. - Key: COUCHDB-647 URL: https://issues.apache.org/jira/browse/COUCHDB-647 Project: CouchDB Issue Type: Bug Affects Versions: 0.9.1, 0.10 Environment: Ubuntu Hardy Reporter: eric casteleijn Priority: Critical Our production server crashed with the following fragment in the error log http://friendpaste.com/3VfsxGrH2XxvkqE3XQA4oy It appears that this is due to a corrupted or zero size database file. Chris Anderson suggested the attached patch to improve the logging in this case. doppler came up with a script that reproduces the crash: touch /Applications/CouchDBX-0.10.1-R13B03-Leopard.app/Contents/Resources/couchdbx-core/couchdb/var/lib/couchdb/test.couch while true ; do curl -X GET http://localhost:5984/test ; done NOTE: On the server that crashed we do not manipulate database files in any other way than calling the REST interface, so it is still a mystery how zero sized dbs get created. I will investigate by digging through the logs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Running out of erlang processes.
Today our production couchdb suddenly crashed and refused to restart,
(scrubbed end of the logfile included.) After asking on #couchdb, Jan
kindly helped me understand that this was due to running into the
default limit of allowed erlang processes (32K and a bit.) and suggested
increasing the number.
While I have no problem doing that if it solves the problem, I am still
left with some questions that I hope you all can help me answer.
I'll first sketch the set-up of our system very globabally, and tell you
what we were doing at the time that might have had an impact:
We have a single server node, with a lot of clients replicating to and
from databases it contains, and a web interface reading from and writing
to it. The server node contains somewhere between 200K and 300K databases.
I was rather naively doing collection some stats from the server by
running something like the following pseudo code:
all_dbs = GET "/_all_dbs"
for db in all_dbs:
sleep for a little bit
GET "/" + db
to get the number of databases and the total number of documents in all
databases. After some minutes of running this, the crash occured. This
might not be related to us running the script, but that would be one
hell of a coincedence, as we haven't seen this particular error before.
Also after aborting the script, couchdb seemed to slowly recuperate.
My questions in declining order of ulcerinducingness:
1. Can anyone explain why the above would cause CouchDB to run out of
processes?
2. Are there more scenarios like this that can cause such crashes?
3. Is there a way to monitor the number of processes in use? (Ideally I
would like to have it in _stats, although I don't know how feasible that
is.)
4. If this is not an outright bug in CouchDB, is there any way to
degrade a little more gracefully in cases like this? It reminds me of
the errors CouchDB throws when running out of file descriptors. I would
things to slow down and maybe time out on connections, rather than cause
crashes, or (in the case of file descriptors) return server errors.
--
- eric casteleijn
https://launchpad.net/~thisfred
http://www.canonical.com
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.23291.1877>] 'HEAD' /u%2F053%2F2a4%2F101325%2Fnotes {1,1}
Headers: [{'Accept',"application/json"},
{'Accept-Encoding',"compress, gzip"},
{'Authorization',"OAuth realm=\"\", oauth_nonce=\"76187806\", oauth_timestamp=\"1257868744\", oauth_consumer_key=\"ubuntuone\", oauth_signature_method=\"HMAC-SHA1\", oauth_version=\"1.0\", oauth_token=\"*\", oauth_signature=\"*\""},
{'Connection',"Keep-Alive"},
{'Host',"couchdb.one.ubuntu.com"},
{'User-Agent',"couchdb-python 0.6"},
{'Via',"1.1 couchdb.one.ubuntu.com"},
{'X-Forwarded-For',"147.102.133.18"},
{"X-Forwarded-Host","couchdb.one.ubuntu.com"},
{"X-Forwarded-Server","couchdb.one.ubuntu.com"},
{"X-Forwarded-Ssl","on"}]
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.23291.1877>] OAuth Params: [{"oauth_nonce","76187806"},
{"oauth_timestamp","1257868744"},
{"oauth_consumer_key","ubuntuone"},
{"oauth_signature_method","HMAC-SHA1"},
{"oauth_version","1.0"},
{"oauth_token","*"},
{"oauth_signature","*"}]
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.23291.1877>] request_group {Pid, Seq} {<0.1222.1877>,105002}
[Tue, 10 Nov 2009 15:58:59 GMT] [error] [emulator] Too many processes
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.11658.1881>] 'GET' /u%2F043%2F3d6%2F128956%2Fcontacts {1,1}
Headers: [{'Accept-Encoding',"identity"},
{'Authorization',"Basic *"},
{'Content-Type',"application/json"},
{'Host',"*:9030"},
{'User-Agent',"couchdb minimal http interface"}]
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.11658.1881>] OAuth Params: []
[Tue, 10 Nov 2009 15:58:59 GMT] [debug] [<0.22836.1877>] 'GET' /u%2Fad1%2F288%2F12708%2Fnotes/_local%2F06b93d053bca1fb7402800b657ba29bd {1,
1}
Headers: [{'Accept',"application/json"},
{'Accept-Encoding',"gzip"},
{&#
Re: Big problems with writes to _config
Jan Lehnardt wrote: On 28 Oct 2009, at 12:09, Jan Lehnardt wrote: HI, On 28 Oct 2009, at 03:46, Adam Kocoloski wrote: On Oct 27, 2009, at 7:49 PM, eric casteleijn wrote: 1) extending or disabling the couch_config gen_server timeout. The default is 5000 milliseconds. This is a one-line patch. […] In each case the response times for PUT/_config/... may become uncomfortably long, but at least you won't be serving 500s from couch. Best, Adam great analysis Alan. Here's the proposed patch for 1) that sets the gen_server timeout to 60 seconds: Thanks to *Adam* (sorry :), this is the more appropriate patch: http://www.friendpaste.com/2ztG8995onyXQGpXrFWZGd Just to close the circle: thanks everyone who replied and tried to help! I filed a bug for the issue, and attached a proposed fix written by my Colleague Gustavo Niemeyer. https://issues.apache.org/jira/browse/COUCHDB-545 I'm a bit busy at the moment, so sorry if I seem a bit curt, I am *very* appreciative of all of your help and ideas! -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
[jira] Updated: (COUCHDB-545) speed of couch_config_writer.erl O(n^2) with size of local.ini
[ https://issues.apache.org/jira/browse/COUCHDB-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn updated COUCHDB-545: Attachment: couch_config_writer.erl > speed of couch_config_writer.erl O(n^2) with size of local.ini > -- > > Key: COUCHDB-545 > URL: https://issues.apache.org/jira/browse/COUCHDB-545 > Project: CouchDB > Issue Type: Bug >Affects Versions: 0.10, 0.10.1, 0.11 > Reporter: eric casteleijn > Fix For: 0.10, 0.10.1, 0.11 > > Attachments: couch_config_writer.erl > > > PUT requests to _config slow down dramatically as the size of local.ini grows. > Attached version of couch_config_writer.erl (written by Gustavo Niemeyer) > takes the write time to the config file from O(n**2) to O(n) resulting in > dramatic speedups when the size of local.ini grows to non trivial sizes. > (change from 5 minutes to 2 seconds observed on our server.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-545) speed of couch_config_writer.erl O(n^2) with size of local.ini
speed of couch_config_writer.erl O(n^2) with size of local.ini -- Key: COUCHDB-545 URL: https://issues.apache.org/jira/browse/COUCHDB-545 Project: CouchDB Issue Type: Bug Affects Versions: 0.10, 0.10.1, 0.11 Reporter: eric casteleijn Fix For: 0.10, 0.10.1, 0.11 PUT requests to _config slow down dramatically as the size of local.ini grows. Attached version of couch_config_writer.erl (written by Gustavo Niemeyer) takes the write time to the config file from O(n**2) to O(n) resulting in dramatic speedups when the size of local.ini grows to non trivial sizes. (change from 5 minutes to 2 seconds observed on our server.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Big problems with writes to _config
eric casteleijn wrote:
eric casteleijn wrote:
We have the following setup:
2 near identical public facing django servers communicating with one
couchdb server. The couchdb server is oauth authenticated and people
can access it directly (well, through an apache proxy) if they have
the tokens to do so. New users are signed up through these django
servers, after which they add the user and their tokens to couchdb.
(the user through a POST to _users and the tokens through PUTs to
_config)
We see this failing a lot, now to the point where we think it fails
all the time (since all those systems have separate logs not all of
which we have access to, this is not trivial to piece together.)
The errors the API servers get back all look like these (the lines
starting with '(500':
'2009-10-27 22:35:15,357 ERRORUbuntuOne.couch: failed to add *
= 40693 to section [oauth_token_users] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_users","*","40693",true}]}'))'
'2009-10-27 22:35:20,399 ERRORUbuntuOne.couch: failed to add *
= * to section [oauth_token_secrets] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_secrets","*",\n
"*",\n true}]}'))'
Corresponding errors in the couchdb.log look like:
Oops, sent before pasting that in:
[Tue, 27 Oct 2009 23:29:42 GMT] [error] [<0.9591.67>] Uncaught error in
HTTP request: {exit,
{timeout,
{gen_server,call,
[couch_config,
{set,"oauth_token_secrets",
"*",
"*",
true}]}}}
[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] Stacktrace:
[{gen_server,call,2},
{couch_httpd_misc_handlers,handle_config_req,1},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[Tue, 27 Oct 2009 23:29:42 GMT] [debug] [<0.9591.67>] httpd 500 error
response:
{"error":"timeout","reason":"{gen_server,call,\n
[couch_config,\n {set,\"oauth_token_secrets\",\"*\",\n
\"*\",\n true}]}"}
[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] 91.189.89.54 - -
'PUT' /_config/oauth_token_secrets/* 500
And another entry I see a *lot* in the couchdb logs, more than the 500s
even is this:
[Tue, 27 Oct 2009 23:29:37 GMT] [debug] [<0.9589.67>] 'PUT'
/_config/oauth_token_secrets/k10rBFRc6Lv20sWWBtSX {1,1}
Headers: [{'Accept',"application/json"},
{'Accept-Encoding',"identity"},
{'Content-Length',"82"},
{'Host',"marang.canonical.com:9030"},
{'User-Agent',"couchdb-python ?"}]
[Tue, 27 Oct 2009 23:29:37 GMT] [debug] [<0.9589.67>] OAuth Params: []
[Tue, 27 Oct 2009 23:29:37 GMT] [debug] [<0.9589.67>] Minor error in
HTTP request: {unauthorized,<<"Authentication required.">>}
[Tue, 27 Oct 2009 23:29:37 GMT] [debug] [<0.9589.67>] Stacktrace:
[{couch_httpd,authenticate_request,2},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[Tue, 27 Oct 2009 23:29:37 GMT] [debug] [<0.9589.67>] httpd 401 error
response:
{"error":"unauthorized","reason":"Authentication required."}
Which I think is just couch telling the client: sorry, you have to
authenticate, but the logs doesn't show any subsequent resolution, just
a lot of these 401s after eachother. (which may be because we retry a
number of times when a request fails.)
Could one of the problems be that (the way we use) couchdb-python does
not always know that it needs to reauthenticate when it gets a 401?
My theory was that these writes to _config fail because the local.ini
is somehow corrupted, but I can't access that file directly (since it
has users' secrets) or copy it to my machine to test this theory, and
helping someone who is allowed to see it look for anything weird is
like searching for the proverbial needle in the haystack: we have lots
of users, and users can have multiple tokens. Add to that the fact
that you cannot ever delete a line from the .ini file (DELETEs against
keys in _config just empty the value and leave a line like 'foo =
Re: Big problems with writes to _config
eric casteleijn wrote:
We have the following setup:
2 near identical public facing django servers communicating with one
couchdb server. The couchdb server is oauth authenticated and people can
access it directly (well, through an apache proxy) if they have the
tokens to do so. New users are signed up through these django servers,
after which they add the user and their tokens to couchdb. (the user
through a POST to _users and the tokens through PUTs to _config)
We see this failing a lot, now to the point where we think it fails all
the time (since all those systems have separate logs not all of which we
have access to, this is not trivial to piece together.)
The errors the API servers get back all look like these (the lines
starting with '(500':
'2009-10-27 22:35:15,357 ERRORUbuntuOne.couch: failed to add * =
40693 to section [oauth_token_users] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_users","*","40693",true}]}'))'
'2009-10-27 22:35:20,399 ERRORUbuntuOne.couch: failed to add * =
* to section [oauth_token_secrets] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_secrets","*",\n
"*",\n true}]}'))'
Corresponding errors in the couchdb.log look like:
Oops, sent before pasting that in:
[Tue, 27 Oct 2009 23:29:42 GMT] [error] [<0.9591.67>] Uncaught error in
HTTP request: {exit,
{timeout,
{gen_server,call,
[couch_config,
{set,"oauth_token_secrets",
"*",
"*",
true}]}}}
[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] Stacktrace:
[{gen_server,call,2},
{couch_httpd_misc_handlers,handle_config_req,1},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[Tue, 27 Oct 2009 23:29:42 GMT] [debug] [<0.9591.67>] httpd 500 error
response:
{"error":"timeout","reason":"{gen_server,call,\n
[couch_config,\n {set,\"oauth_token_secrets\",\"*\",\n
\"*\",\n true}]}"}
[Tue, 27 Oct 2009 23:29:42 GMT] [info] [<0.9591.67>] 91.189.89.54 - -
'PUT' /_config/oauth_token_secrets/* 500
My theory was that these writes to _config fail because the local.ini is
somehow corrupted, but I can't access that file directly (since it has
users' secrets) or copy it to my machine to test this theory, and
helping someone who is allowed to see it look for anything weird is like
searching for the proverbial needle in the haystack: we have lots of
users, and users can have multiple tokens. Add to that the fact that you
cannot ever delete a line from the .ini file (DELETEs against keys in
_config just empty the value and leave a line like 'foo = \n'!
After speaking to Jan on the channel he proposed that it may be that the
gen_server message inbox overflows and the gen_server times out.
Could that be, under high load, and how can we solve this? Can we
increase the size of this inbox, or can we possibly have multiple
processes handling the access? Whether it's high load or corruption or
something else again, right now it looks like NO new tokens can be
added, and hence no new users can use our system. In short: HALP!
--
- eric casteleijn
https://launchpad.net/~thisfred
http://www.canonical.com
Big problems with writes to _config
We have the following setup:
2 near identical public facing django servers communicating with one
couchdb server. The couchdb server is oauth authenticated and people can
access it directly (well, through an apache proxy) if they have the
tokens to do so. New users are signed up through these django servers,
after which they add the user and their tokens to couchdb. (the user
through a POST to _users and the tokens through PUTs to _config)
We see this failing a lot, now to the point where we think it fails all
the time (since all those systems have separate logs not all of which we
have access to, this is not trivial to piece together.)
The errors the API servers get back all look like these (the lines
starting with '(500':
'2009-10-27 22:35:15,357 ERRORUbuntuOne.couch: failed to add * =
40693 to section [oauth_token_users] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_users","*","40693",true}]}'))'
'2009-10-27 22:35:20,399 ERRORUbuntuOne.couch: failed to add * =
* to section [oauth_token_secrets] of local.ini:
(500, (u'timeout', u'{gen_server,call,\n[couch_config,\n
{set,"oauth_token_secrets","*",\n
"*",\n true}]}'))'
Corresponding errors in the couchdb.log look like:
My theory was that these writes to _config fail because the local.ini is
somehow corrupted, but I can't access that file directly (since it has
users' secrets) or copy it to my machine to test this theory, and
helping someone who is allowed to see it look for anything weird is like
searching for the proverbial needle in the haystack: we have lots of
users, and users can have multiple tokens. Add to that the fact that you
cannot ever delete a line from the .ini file (DELETEs against keys in
_config just empty the value and leave a line like 'foo = \n'!
After speaking to Jan on the channel he proposed that it may be that the
gen_server message inbox overflows and the gen_server times out.
Could that be, under high load, and how can we solve this? Can we
increase the size of this inbox, or can we possibly have multiple
processes handling the access? Whether it's high load or corruption or
something else again, right now it looks like NO new tokens can be
added, and hence no new users can use our system. In short: HALP!
--
- eric casteleijn
https://launchpad.net/~thisfred
http://www.canonical.com
[jira] Commented: (COUCHDB-540) oauth_token_users seems to be in the wrong place
[ https://issues.apache.org/jira/browse/COUCHDB-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769262#action_12769262 ] eric casteleijn commented on COUCHDB-540: - The /users database would be an excellent place to store these, but the current implementation does not do that yet. A separate local_users.ini would be an improvement, but a minor one IMO. > oauth_token_users seems to be in the wrong place > > > Key: COUCHDB-540 > URL: https://issues.apache.org/jira/browse/COUCHDB-540 > Project: CouchDB > Issue Type: Bug > Components: Infrastructure >Affects Versions: 0.10 > Environment: Ubuntu Linux >Reporter: Chris Jones > > If one has many users, the oath_token_users section of /etc/couchdb/local.ini > becomes quite large. This: > a) makes legitimate configuration of the server less friendly > b) makes the file extremely hard to manage via version control > c) is probably a violation of the FHS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-540) oauth_token_users seems to be in the wrong place
[ https://issues.apache.org/jira/browse/COUCHDB-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769249#action_12769249 ] eric casteleijn commented on COUCHDB-540: - I can confirm this is problematic for large numbers of users. The same goes for oauth_token_secrets. > oauth_token_users seems to be in the wrong place > > > Key: COUCHDB-540 > URL: https://issues.apache.org/jira/browse/COUCHDB-540 > Project: CouchDB > Issue Type: Bug > Components: Infrastructure >Affects Versions: 0.10 > Environment: Ubuntu Linux >Reporter: Chris Jones > > If one has many users, the oath_token_users section of /etc/couchdb/local.ini > becomes quite large. This: > a) makes legitimate configuration of the server less friendly > b) makes the file extremely hard to manage via version control > c) is probably a violation of the FHS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [VOTE] Apache CouchDB 0.10.0 release, second round
Noah Slater wrote: Hello, I would like call a vote for the Apache CouchDB 0.10.0 release, second round. +1 Ubuntu 09.10beta (Karmic) 55 of 55 test(s) run, 0 failures (36202 ms) -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
Re: Recent fixes targeted at 0.10.x
Noah Slater wrote: Noah "Awesome" Slater, Consider yourself pinged. w0rd This is by no means meant to apply any extra pressure, but for Ubuntu Karmic we have to package the CouchDB that is going to ship with the final release more or less now. On the other hand we'd love to have an officially released version in there rather than a packaged svn snapshot, since that will make things like backporting critical bugfixes and reporting bugs upstream a lot easier to streamline. Is it realistic that the release will happen this week? "No", is an acceptable answer, in that case we'll just start packaging what's there now. -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
Validation functions and design documents
I was unpleasantly surprised today when I tried to implement a rather critical piece of our infrastructure: the validation to prevent replication of certain design documents which are run on our server node(s). It appears that validation functions don't get called at all when a design document is updated. I used the logging example from the book: http://books.couchdb.org/relax/reference/security#Update%20Validations%20Again and this logs an entry for each update to a regular document, but is eerily silent on updates to design documents. Is there maybe a switch I can turn on in the .ini file, or a different kind of validator I can use for this purpose? -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
Strange problems when PUTting to _config
[mistakenly sent to user@ previously]
(This is on a snapshot of the 0.10 branch and may be fixed already.
Since we're seeing this on a production server and not our development
environments, it's hard to write a reliably failing test.)
After a lot of successful (after a fashion[*]) PUTting to and DELETEing
from _config, we're seeing a sudden error in the logs, after which all
subsequent ones seem to fail. The first failure looks like this
(potential security sensitive info *ed, but looks similar to earlier
successful attempts):
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] 'PUT'
/_config/admins/50415 {1,1}
Headers: [{'Accept',"application/json"},
{'Accept-Encoding',"identity"},
{'Authorization',"OAuth realm=\"\", oauth_nonce=\"*\",
oauth_timestamp=\"1253842863\", oauth_consumer_key=\"*\",
oauth_signature_method=\"HMAC-SHA1\", oauth_version=\"1.0\",
oauth_token=\"*\", oauth_signature=\"*\""},
{'Content-Length',"10"},
{'Host',"raspberry.canonical.com:9030"},
{'User-Agent',"couchdb-python ?"}]
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] OAuth Params:
[{"oauth_nonce","03532062"},
{"oauth_timestamp","1253842863"},
{"oauth_consumer_key","*"},
{"oauth_signature_method","HMAC-SHA1"},
{"oauth_version","1.0"},
{"oauth_token","*"},
{"oauth_signature","*"}]
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.44.0>] saving to file
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
"password"}'
[Fri, 25 Sep 2009 01:41:08 GMT] [error] [<0.13623.4>] Uncaught error in
HTTP request: {exit,
{timeout,
{gen_server,call,
[couch_config,
{set,"admins","50415","password",true}]}}}
[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] Stacktrace:
[{gen_server,call,2},
{couch_httpd_misc_handlers,handle_config_req,1},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.13623.4>] httpd 500 error
response:
{"error":"timeout","reason":"{gen_server,call,[couch_config,{set,\"admins\",\"50415\",\"password\",true}]}"}
[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] 91.189.90.36 - -
'PUT' /_config/admins/50415 500
[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.44.0>] saving to file
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
"-hashed-*"}'
NOTE: We get back a 500 (why, I don't know) and after that, the hashed
password is happily written to the file anyway?! (Note also: for
succesful writes, the order is the same, at least in the log file, which
seems wrong: it sends back a 200 OK and then attempts the write to the
.ini file...)
After this, all PUTS to _config seem to fail. Unfortunately it appears
that one of my colleagues in their haste to fix this problem, has
removed the local.ini file and started with a fresh one, so it is
impossible for me to determine what state it was in. (I have asked an
admin who is allowed to look in those files, but there seems to be
nothing strange there now, and the admin user that was being PUT does
not appear there.)
[*] Succesful in that we have to keep trying until we succeed, sometimes
several hundred times. I think this is because some erlang bits need to
be restarted after each write, like the auth_handler? Perhaps that is
what failed to come back up at all after above mishap...
I've been seeing errors like this which seem to suggest that, but I suck
at reading erlang/couchdb tracebacks:
[Fri, 25 Sep 2009 06:52:16 GMT] [error] [<0.3054.5>] {error_report,<0.24.0>,
{<0.3054.5>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.3054.5>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_config,
{get,"httpd","authentication_handlers",undefined}]}},
[{gen_server,call,2},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.63.0>,#Port<0.42195>]},
{dictionary,[{jsonp,undefined}]},
{trap_exit,false},
{status,running},
{heap_size,6765},
{stack_size,24},
{reductions,14822}],
[]]}}
--
- eric casteleijn
http://thisfred.blogspot.com
Re: [VOTE] Apache CouchDB 0.10.0 release
Noah Slater wrote: Hello, I would like call a vote for the Apache CouchDB 0.10.0 release. This release added ETap sources from Nick Gerakines under the BSD license, We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release, so get stuck in! We are voting on the following release artifacts: http://people.apache.org/~nslater/dist/0.10.0 These artifacts have been built from the 0.10.0 tag in Subversion: http://svn.apache.org/repos/asf/couchdb/tags/0.10.0 Happy voting, Noah +1 built[*] and passed all tests on Ubuntu 9.10 (Karmic) alpha. [*] I have to configure *and* run as follows, or the view server does not work at all, but that's a problem with xulrunner-dev, rather than with couchdb, I gather: ./configure --with-js-lib=/usr/lib/xulrunner-devel-1.9.1.3/lib --with-js-include=/usr/lib/xulrunner-devel-1.9.1.3/include LD_LIBRARY_PATH=/usr/lib/xulrunner-devel-1.9.1.3/lib utils/run -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
Re: Per-DB Auth Ideas and Proposal
I played around with read & write access lists but write can be handled by the validation function so it's not necessary. I'm not entirely convinced the current validation functions, living in more or less as content in the database are sufficient for anything but very simple permission models. I foresee a lot of problems if two users have very broad permissions (say what is currently known as 'admin') in the same database, but one shouldn't be able to modify certain design documents. Any way of modelling that with validation functions seems very tricky to me: User 1, whom we shall call 'super_admin', creates a design document, with a validator function restricting all write access to any document with id prefix '_design/special_view_' to user 'super_admin'. The validator function itself should not be modifiable by user 2 'regular_admin' or it is worthless, of course. This would work, except there is nothing stopping 'regular_admin' from adding their own validator, restricting 'super_admin' similarly from those design_documents, and from the design_document this validator is in, and now everybody is locked out. Currently, I see no other possibility to model this (very real) use case, where you want a user to be able to write to almost every document (by replication from their machine), except for a few views (because we'll be executing those on our cloud server node) without ending up with a variant on the above cold war arms race scenario. The simplest way to solve this that I see is if the permissions live in a place that regular_admin simply does not have any access to (i.e. the global _user db) but where a global admin can grant or restrict permissions at will. -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
[jira] Commented: (COUCHDB-498) .ini file chaining not properly reset by -n switch
[ https://issues.apache.org/jira/browse/COUCHDB-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753322#action_12753322 ] eric casteleijn commented on COUCHDB-498: - This fixes the issue for me on the 0.10 branch, thanks so much, Adam! > .ini file chaining not properly reset by -n switch > -- > > Key: COUCHDB-498 > URL: https://issues.apache.org/jira/browse/COUCHDB-498 > Project: CouchDB > Issue Type: Bug >Affects Versions: 0.10 > Environment: Ubuntu Linux 09.10 (karmic) unreleased > Reporter: eric casteleijn > Fix For: 0.10 > > Attachments: background_start_config_reset.patch > > > I've added as line 220 in /usr/bin/couchdb: > echo "startarguments: $start_arguments" > Now, when I do: > /usr/bin/couchdb -n -a > /tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini -p > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.pid -o > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stdout -e > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stderr -b > I get: > Apache CouchDB has started, time to relax. > e...@thelog:~$ startarguments: \"/etc/couchdb/default.ini\", > \"/etc/couchdb/local.ini\", \""/etc/couchdb/default.ini"\", > \""/tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini"\" > which suggests to me that the -n switch in this case does not actually reset > the .ini chain. > This came to light when I noticed that any admin users I added in my system > CouchDB also affected any other CouchDB I ran on the same machine. Which > seems logical if my analysis of the bug is correct, since the admin users > would be stored in /etc/couchdb/local.ini which is always picked up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-498) .ini file chaining not properly reset by -n switch
[ https://issues.apache.org/jira/browse/COUCHDB-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752724#action_12752724 ] eric casteleijn commented on COUCHDB-498: - I manually applied the two changes it as best I could against my installed couch (by editing /usr/bin/couchdb directly, probably not the smartest idea) but that showed zero difference. Can you confirm that "-n -a some.ini" really completely bypasses all the .ini files in /etc/couchdb/ ? > .ini file chaining not properly reset by -n switch > -- > > Key: COUCHDB-498 > URL: https://issues.apache.org/jira/browse/COUCHDB-498 > Project: CouchDB > Issue Type: Bug >Affects Versions: 0.10 > Environment: Ubuntu Linux 09.10 (karmic) unreleased >Reporter: eric casteleijn > Fix For: 0.10 > > Attachments: background_start_config_reset.patch > > > I've added as line 220 in /usr/bin/couchdb: > echo "startarguments: $start_arguments" > Now, when I do: > /usr/bin/couchdb -n -a > /tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini -p > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.pid -o > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stdout -e > /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stderr -b > I get: > Apache CouchDB has started, time to relax. > e...@thelog:~$ startarguments: \"/etc/couchdb/default.ini\", > \"/etc/couchdb/local.ini\", \""/etc/couchdb/default.ini"\", > \""/tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini"\" > which suggests to me that the -n switch in this case does not actually reset > the .ini chain. > This came to light when I noticed that any admin users I added in my system > CouchDB also affected any other CouchDB I ran on the same machine. Which > seems logical if my analysis of the bug is correct, since the admin users > would be stored in /etc/couchdb/local.ini which is always picked up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-498) .ini file chaining not properly reset by -n switch
.ini file chaining not properly reset by -n switch -- Key: COUCHDB-498 URL: https://issues.apache.org/jira/browse/COUCHDB-498 Project: CouchDB Issue Type: Bug Affects Versions: 0.10 Environment: Ubuntu Linux 09.10 (karmic) unreleased Reporter: eric casteleijn Fix For: 0.10 I've added as line 220 in /usr/bin/couchdb: echo "startarguments: $start_arguments" Now, when I do: /usr/bin/couchdb -n -a /tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini -p /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.pid -o /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stdout -e /tmp/tmp871WkT/xdg_cache/desktop-couch/desktop-couchdb.stderr -b I get: Apache CouchDB has started, time to relax. e...@thelog:~$ startarguments: \"/etc/couchdb/default.ini\", \"/etc/couchdb/local.ini\", \""/etc/couchdb/default.ini"\", \""/tmp/tmp871WkT/xdg_config/desktop-couch/desktop-couchdb.ini"\" which suggests to me that the -n switch in this case does not actually reset the .ini chain. This came to light when I noticed that any admin users I added in my system CouchDB also affected any other CouchDB I ran on the same machine. Which seems logical if my analysis of the bug is correct, since the admin users would be stored in /etc/couchdb/local.ini which is always picked up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-491) Support replication over SSL
Support replication over SSL Key: COUCHDB-491 URL: https://issues.apache.org/jira/browse/COUCHDB-491 Project: CouchDB Issue Type: Improvement Reporter: eric casteleijn Fix For: 0.10 CouchDB currently does not support replication over SSL which is a problem when replicating non public databases between two CouchDB servers over the internet. It seems SSL support is almost there, though: If ssl is started from bin/couchdb, push replication to a couchdb that is behind an SSL proxy works (i.e. to an https:// url), but pull replication from same fails, because apparently the request is not encrypted at all. (At least nginx seemed to think it wasn't.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: First iteration on Authentication and Authorization Wiki
Curt Arnold wrote: I've spent some time trying to collect resources related to authentication and authorization in CouchDB and to try to get my mind around what is currently in CouchDB. Would appreciate any feedback, comments or corrections. Will try to start on some use cases tomorrow. http://wiki.apache.org/couchdb/Authentication_and_Authorization I've added an example OAuth session to the oauth authentication section. All feedback on that very welcome! -- - eric casteleijn https://launchpad.net/~thisfred http://www.canonical.com
[jira] Created: (COUCHDB-438) Add per database (OAuth) authentication to couchdb
Add per database (OAuth) authentication to couchdb -- Key: COUCHDB-438 URL: https://issues.apache.org/jira/browse/COUCHDB-438 Project: CouchDB Issue Type: New Feature Affects Versions: 0.10 Reporter: eric casteleijn Fix For: 0.10 In set-ups where a couchdb node holds databases for many different users that are not allowed to see each other's data, it is vital that it is possible to give users separate roles for each database. It would be best if adding new users, and assigning roles to users for specific databases could be done through the http interface, as modifying .ini files becomes unwieldy for large numbers of users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-420) OAuth authentication support (2-legged initially) and cookie-based authentication
[
https://issues.apache.org/jira/browse/COUCHDB-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736012#action_12736012
]
eric casteleijn commented on COUCHDB-420:
-
Great work, Jason!
I've gotten the newest version of the patch working (which I'm sure will be
attached soon.) Getting things to work still required a lot of hand holding, so
I would love to have (and help write) some detailed setup documentation. In
particular:
1. An example oauth.ini and an elaborate description of all the steps it takes
to authenticate against that.
2. An example replication session from the command line (using curl, or python
+ python-oauth or somesuch.) using the credentials created in 1.
3. A description of the process to add new (oauth) authenticated users to a
running couchdb. (If that is currently possible, I think Jan suggested it was.)
> OAuth authentication support (2-legged initially) and cookie-based
> authentication
> -
>
> Key: COUCHDB-420
> URL: https://issues.apache.org/jira/browse/COUCHDB-420
> Project: CouchDB
> Issue Type: New Feature
> Components: HTTP Interface
>Reporter: Jason Davies
> Fix For: 0.10
>
> Attachments: oauth.1.diff
>
>
> This patch adds two-legged OAuth support to CouchDB.
> 1. In order to do this, a couple of changes have been made to the way auth
> handlers are used. Essentially, the patch allows multiple handlers to be
> specified in a comma-separated list in the following in the [httpd] section
> of your .ini config e.g.
> authentication_handlers = {couch_httpd_oauth,
> oauth_authentication_handler}, {couch_httpd_auth,
> default_authentication_handler}
> The handlers are tried in order until one of them successfully authenticates
> and sets user_ctx on the request. Then the request is passed to the main
> handler.
> 2. Now for the OAuth consumer keys and secrets: as Ubuntu need to be able to
> bootstrap this i.e. add tokens without a running CouchDB, I have advised
> creating a new config file in $PREFIX/etc/couchdb/default.d/ called oauth.ini
> or similar. This should get read by CouchDB's startup script when it loads
> its config files (e.g. default.ini and local.ini as well). There are three
> sections available:
> i. [oauth_consumer_secrets] =
> ii. [oauth_token_secrets] =
> iii. [oauth_token_users] =
> The format I've used above is [section name] followed by how the keys and
> values for that section will look on subsequent lines. The secrets are a way
> for the consumer to prove that it owns the corresponding consumer key or
> access token. The mapping of auth tokens to usernames is a way to specify
> which user/roles to give to a consumer with a given access token.
> In the future we will also store tokens in the user database (see below).
> 3. OAuth replication. I've extended the JSON sent via POST when initiating a
> replication as follows:
> {
> source: {
>url: ,
>auth: {
> oauth: {
>consumer_key: ,
>consumer_secret: ,
>token_secret: ,
>token:
> }
>}
> },
> target: /* same syntax as source, or string for a URL with no auth info, or
> string for local database name */
> }
> 4. This patch also includes cookie-authentication support to CouchDB. I've
> covered this here:
> http://www.jasondavies.com/blog/2009/05/27/secure-cookie-authentication-couchdb/
> The cookie-authentication branch is being used on a couple of live sites and
> the branch has also been worked on by jchris and benoitc. As well as cookie
> auth it includes the beginnings of support for a per-node user database, with
> APIs for creating/deleting users etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-422) PUT to _local/doc_id creates document '_local'
PUT to _local/doc_id creates document '_local'
--
Key: COUCHDB-422
URL: https://issues.apache.org/jira/browse/COUCHDB-422
Project: CouchDB
Issue Type: Bug
Affects Versions: 0.10
Environment: Ubuntu 9.04
Reporter: eric casteleijn
Priority: Minor
Fix For: 0.10
After davisp's revision r796246 doing a put to a document id like
'_local/doc_id' results in a visible, and apparently non-local document with id
'_local'. When escaping the slash as '%2F' everything works as before, and
expected, i.e. a local document with the above id is created.
To test:
curl -X PUT -d '{"foo": "bar"}' http://127.0.0.1:5987/db1/_local/yokal
result:
{"ok":true,"id":"_local","rev":"1-770307fe8d4210bab8ec65c59983e03c"}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-360) Add global _changes feature
[ https://issues.apache.org/jira/browse/COUCHDB-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733320#action_12733320 ] eric casteleijn commented on COUCHDB-360: - I would really like to have a global _changes feed, as I anticipate having a lot of databases to deal with in a similar manner. One question though: Is it impossible, inefficient to include more details of the changes? I would really like to at least optionally have the same information available as the per db _changes feed exposes. > Add global _changes feature > --- > > Key: COUCHDB-360 > URL: https://issues.apache.org/jira/browse/COUCHDB-360 > Project: CouchDB > Issue Type: New Feature >Reporter: Robert Newson > Attachments: global_changes.patch > > > Add a global version of _changes to allow monitoring of changes to all > databases. It does not need to include details of the change as clients of > this interface must track update sequences for each database anyway. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: _changes resource
Adam Kocoloski wrote: On Jul 6, 2009, at 3:58 PM, Chris Anderson wrote: == Deleted and Conflicts== _all_docs_by_seq includes a 'deleted' flag and a list of 'conflicts'. Should the _changes API to do the same? The plan is to drive replication from changes, so anything needed by replication is on the roadmap. I don't think it'd hurt to have any of those but Damien would be better to answer this one. The deleted=true flag probably won't be needed by the replicator, because the _changes feed includes the deletion revid. I expect that the replicator will just download this revision like any other, find the _deleted:true bit set in the document, and delete the document on the target. Note that replication is an important user of _changes, but by no means the only one, if update notifiers go away. (Which I think everyone agrees would be a good idea.) I would like to have the option to not only see when a document was deleted, but in addition when one was first created on the node in question, which in my application would require special action, over and above what needs to happen for an update to a document. _conflicts and _deleted_conflicts are more interesting. When one of these occurs, the document shows up in the _changes feed, but the revision in that row is the latest revision of the document, not the conflict/deleted_conflict rev. Unlike _all_docs_by_seq, it's not possible for the replicator to determine the list of revisions to replicate solely by analyzing the _changes feed. I think the most efficient solution is to start including conflict and deleted_conflict revisions in the revlist in the _changes row. I don't know the revision tree well enough to know if it's possible to identify the set of all conflict revisions that were saved after update_sequence N, but if it is that would be a neat restriction. This sounds like a good idea. Another option might be to configure a metadata-only request so that the replicator could check what revisions exist on the source for each updated document. Could be a useful thing to have in general. And so does this. What I would like to add: Right now the _changes feeds are per db, and while that is great in some use cases (like replication.) In others, where there are many thousands of databases, one global _changes feed would be much more practical. It is also how the current update notifiers work, so not having this option would break existing applications, at least it would mine. ;) One last thing that would be great to have is a way to configure what information goes into a particular _changes feed or an option to write your own _changes-like feeds in javascript, like you could a view, so that one could have a feed of changes to the values of a particular field, for instance. It would make it so that processes that act on updates from the db never have to query back into the db for additional data, which could be a performance win. (That is if the configurable _changes feeds aren't too much of a performance loss.) I've also filed a feature request in JIRA with these suggestions: https://issues.apache.org/jira/browse/COUCHDB-390 but more discussion, here or on that issue, is most welcome. -- - eric casteleijn http://www.canonical.com
Re: "Personal" Couch DB.
thadgui...@gmail.com wrote: Eric, I like the idea of OAuth incorporated down the line. Also, It seems as if Brian and yourself are talking more in general about a Multi-Tenancy environment within a single CouchDB or is it not as grand of an idea yet ? ( "Are we there yet?" ) I am not sure that multi-tenancy is exactly what we're aiming for, or even that couchdb itself neatly fits into that terminology, but I'll try to describe what we are thinking of doing with couchdb: Our use case is to have a single couchdb server per user, and (potentially) many databases for that user within each such server. They will be running on the end user's system. We intend to replicate all of this data between the cloud and all of a user's registered machines, but only as a means of backup, universal accessibility (i.e. from the user's work computer and potentially through the web.) and as a means of sharing pieces of data with other users of the service. If I understand multi-tenancy correctly, that would mean leasing out couch instances on our hardware, which is only half the story at best: I think it's important to stress that *all* the data that is in the cloud couch instance will be in the users desktop instance as well, and they are free to do what they want with it. In fact they will be able to use the same software to replicate between computers on their LAN if they do not choose to use our online service at all, or keep using it without any data loss when they decide to stop using it. -- - eric casteleijn http://www.canonical.com
Re: "Personal" Couch DB.
Brian Candler wrote: On Wed, Jun 24, 2009 at 07:55:36AM -0400, Elliot Murphy wrote: In fact, this effort to run a CouchDB for every single user account on a computer Interesting. A few questions: Hi Brian, Elliot's on holiday, so I'm going to be presumptious and respond in his stead. I hope I don't say anything too far from the truth. (1) What stops user A from accessing user B's data? Will this be done with HTTP authentication? Perhaps in this context it might make more sense to use a Unix domain socket, but then again, nobody talks HTTP over those :-( We've thought about using sockets, and while that would be a good way to do it, it would also mean a lot of work making HTTP client libraries work with that. We think this could be more easily solved by having authorization in CouchDB, and we're talking with Jan and Chris about adding OAuth to CouchDB. OAuth is a good candidate since it's explicitly meant for authenticating applications with servers over HTTP, so it's a good fit for our use case, where the server happens to live on the same machine, but also a good fit for talking to CouchDB on a remote server. (2) Is the per-user Erlang/CouchDB process going to be started when the user logs into the graphical desktop (gdm/kdm?) What about having a single system-wide CouchDB instance instead of spawning a new Erlang VM for each user? Then at least, if I log into my home machine via ssh (while not logged into the console), the service would still be there. For Ubuntu, the per-user process will be started up by being prompted from a DBUS service. We're making a DBUS service that handles getting OAuth credentials from the Gnome keyring and ensuring that the user's CouchDB is running. We thought about having a system-wide CouchDB, and having separate databases for each user. This is hard, though, because we want to have the database files be in each user's home directory, so that they get backed-up and so on in a way that users expect. We also don't want to run CouchDB as root. So, we're setting things up with one CouchDB server per user. It would be easy to set up CouchDB to be running all the time, for users who want that. We need to make sure it doesn't affect the whole system's start-up and log in time, though, so that makes starting CouchDB on demand more attractive. Hope this helps, -- - eric casteleijn http://www.canonical.com
[jira] Created: (COUCHDB-390) proposed additions to _changes feed
proposed additions to _changes feed --- Key: COUCHDB-390 URL: https://issues.apache.org/jira/browse/COUCHDB-390 Project: CouchDB Issue Type: New Feature Reporter: eric casteleijn The _changes comet feed as a replacement for the update notifications is a great step forward, but I'm currently missing these features: 1. The feeds are now per database, rather than per server, which is nice in some use cases, but in others it's preferable to have a server wide feed (In the case of many thousands of databases on a server, and a single process reacting to updates, by putting messages on a message queue for instance). Ideally I'd like to have both server wide and per database _changes feeds available. 2. I'd like to have a configurable _changes feed that let's me declare what data will be output in the update rows in the feed, in a similar way to writing views. 3. I'd like to (optionally) have information on whether a particular update was a document creation, update or deletion, so clients would be able to react to those differently without needing to query into the database to find out what happened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Document update notifications
Chris Anderson wrote: The next step on the roadmap is to give the ability to filter the comet notifications with a JavaScript function. This code should not be too hard to write but I doubt I'll have time to get to it this week. There are a few other people who are capable of adding the function, if any of them add the capability I'll be happy to review and apply the patch. Otherwise it should just be a matter of time before the comet feature is complete. There are other parts of the story we could add but haven't. For instance, it should be possible to do a long poll on a particular document URL if you are waiting for it to be modified. Also, replication should be able to run against the _changes API instead of _all_docs_by_seq, although I'm not sure that's crucial for your use case. I hope that give enough clarity for you. That is very good news! I did not know that the _changes view was already in trunk. I'll get it up and running and see if I can change our code to start listening to it. When you talk about filtering, does that mean taking the output it currently gives and suppressing lines (which contain seq, doc_id and list of revisions, if I understand correctly) or to allow emitting other fields, or depending on the values of other fields as well? The use case I am thinking of again is to be able to distinguish between creation, modification and deletion of documents. Perhaps the first could be done by inspecting revision numbers, although that may be relying on internals too much, and I don't know what could be done to detect whether a document was deleted. The alternative I have seen suggested a lot, to have separate documents that record deletion/creation events is hard to implement when the code accessing the database is not the same on all nodes (i.e. you have no control over some nodes, but would still like to have an idea of when documents that were created there were first seen/deleted by the node you do control.) Also it concerns me that the _changes feed is now per database, rather than per server, (although there is https://issues.apache.org/jira/browse/COUCHDB-360 which implies a global _changes feed, but it seems that that in turn will only include the database name.) Having to listen to one feed per database seems like it could take up a lot of resources if one has a very large number of databases, which I imagine we will. Having to constantly query the databases for more information after each update also seems like it would become a performance burden very soon. This is one of the main reasons (the other being couchdb running asynchronous, and hence hard to test, processes) that I am looking forward to seeing the end of the update triggers as they are. Ideally, I would like to have one _changes feed per server, with configurable output, so that any external processing that needs to happen can happen by one process per server, that will not have to query into the database after every received update. -- - eric casteleijn http://www.canonical.com
Re: Document update notifications
Would the comet notifications supersede the (currently implemented) db update notifications as well? Are there any estimates as to when that would come around. Pretty sure the idea is to replace the current system's functionality. I'm not sure if that means update_notification will go away or not, but from what I gather on the expected use cases it'll be a "new code should use comet" type of situation. As far as estimates, I'm not sure. I don't have too much knowledge on the current status, just that there was work being done on some necessary bits. Keep an eye on dev@ and you'll be up to date as things start building up. I'm working on it and even I can't say when :) Too much stuff pops up that's out of my control, so I don't give completion estimates any more. But I will say it's my highest priority development task right now. I'm sorry for asking again, but this feature (or rather, getting rid of the update notifiers) is turning out to be more and more critical to us. Is there *anything* we can do to help this along? If so, please let me know, on or off list. -- - eric casteleijn http://www.canonical.com
Re: couch sequence numbers and _all_docs_by_seq
If I remember correctly your problem, you want to process documents when they first cross the transom into your cloud. If they are then replicated out and about and come back again later, you don't need to reprocess them. Of course anything involving client-changeable data is not a 100% guarantee, but if you can live with occasionally reprocessing documents, then you might try something like: A view of all docs which do not have the cloud_processed property. And then a process which is always trying to keep that view empty by processing the docs it lists in whatever manner you need. This is exactly what we do now: there is an update trigger, that queries for unprocessed documents, and bulk updates them to have a 'first_seen' field that holds the sequence number that the database was at when the trigger was fired. The reason I would like to have access to the sequence number of documents in my views is similar but different: It would allow me to write a view that gets all the documents of a particular type that were last updated between two sequence numbers, without relying on an id prefix, which feels awkward, and is problematic for us, since we have URIs for document types, which obviously won't work as part of the id, so we'd have to keep a mapping from type to prefix, and that is another step away from simplicity. Of course you'll be relying on clients to trigger "download" (from the cloud to their local) replication about as often as they trigger "upload" replication, otherwise your process will start to stack up docs in a conflict state. The other solution I think we talked about was maintaining an independent database in the cloud, which just tracks which document-ids have been processed. This avoids the conflicts scenario, and when you think about what it means to the disks, it's about the same cost as maintaining that view. However, you end up querying it over and over again for each document you see, instead of just seeing the relevant docs. That is still a solution we might have to choose, but even if it's not a performance problem, it increases code complexity. I'd do whatever possible to avoid recording update_seq at the application level, as CouchDB is not designed to make multi-node guarantees about that property. Yes, that way madness lies, and I'm not suggesting that. All I'd like is for views I create myself to be able to use _seq as a key (or possibly value) like _all_docs_by_seq does, to have more efficient ways of querying for data that changed either within the node or through replication, i.e. with a single view, rather than through calling _all_docs_by_seq and filtering in application code. -- - eric casteleijn http://www.canonical.com
Re: couch sequence numbers and _all_docs_by_seq
Ok, but just so we're on the same page, I think Eric was talking about using the update_seq for that document in the view (i.e., the key for that document in _all_docs_by_seq), not the server's latest update_seq. I don't think that usage violates any requirements of the map function. Yes that is correct. Adam I'd have to think harder on it. I'm pretty sure that you'd end up with the same documents and different view output depending on the node you were using. I think so too: At least if that would be the case for the _all_docs_by_seq, and I'm pretty sure it is. That is not a problem for my use case, where we have only one particular node that will be running the views for which I;d want to use the _seq. You could almost fix it by transmitting 'dead' update_seq's but that's a long road for the immediate question. If I understand you correctly, that's partly what I'm doing now as a solution to a different problem, but it's really the opposite of elegant: To have a handle on when a document was first seen on the 'node of interest' we have an update trigger write a sequence number (which is approximate to the one the db was at on the when the document was first seen there) into a 'first_seen' field. Doing the same for a 'last_modified' field would be worse, or even impossible, since writing to that field would trigger the update trigger again. -- - eric casteleijn http://www.canonical.com
Re: couch sequence numbers and _all_docs_by_seq
Paul Davis wrote: Eric, Implementation isn't a problem, but I would fear that replication could screw views up. As in, your views would depend on the order of document modification which could be impacted quite heavily by replication etc. Ah right, hadn't thought of that. So in a system using replication, views that would use the sequence number as a key would get a severe performance penalty? (Just checking to see if I understand the implications.) That might be a good enough reason not to expose this as a feature, lest people start depending on it and saying CouchDb is slow as a result. If this is true, I suppose it also holds for the built in _all_docs_by_seq view? -- - eric casteleijn http://www.canonical.com
couch sequence numbers and _all_docs_by_seq
Is there any reason not to expose sequence numbers to any view other than '_all_docs_by_seq'? I find myself wishing I had access to them again and again. A simple use case is to be able to query for documents of a particular type that were last updated within a range of sequence numbers. Currently I can only do this if I have separate databases for each type, or id prefixes, so that I can filter out the ids I don't want out of the views results that way. If I could build by own view that was able to emit the sequence number as a key, that would make things much more pleasant. Are there compelling reasons against a patch that would make this possible? For instance, would it introduce efficiency penalties to those not using sequence numbers, or would it just be a very hard thing to implement? -- - eric casteleijn http://www.canonical.com
[jira] Commented: (COUCHDB-194) [startkey, endkey[: provide a right-open range selection method
[
https://issues.apache.org/jira/browse/COUCHDB-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696651#action_12696651
]
eric casteleijn commented on COUCHDB-194:
-
Worth noting:
The builtin "_all_docs_by_seq" view right now returns everything with key:
startkey < key <= endkey
I would like to know what the intended consistent way will be, so that I can
already make my higher level API work that way, and not ahve to break clients'
code when this gets changed/fixed.
> [startkey, endkey[: provide a right-open range selection method
> ---
>
> Key: COUCHDB-194
> URL: https://issues.apache.org/jira/browse/COUCHDB-194
> Project: CouchDB
> Issue Type: Improvement
> Components: HTTP Interface
>Affects Versions: 0.10
>Reporter: Maximillian Dornseif
>Priority: Blocker
> Fix For: 0.10
>
>
> While writing something about using CouchDB I came across the issue of "slice
> indexes" (called startkey and endkey in CouchDB lingo).
> I found no exact definition of startkey and endkey anywhere in the
> documentation. Testing reveals that access on _all_docs and on views
> documents are retuned in the interval
> [startkey, endkey] = (startkey <= k <= endkey).
> I don't know if this was a conscious design decision. But I like to promote a
> slightly different interpretation (and thus API change):
> [startkey, endkey[ = (startkey <= k < endkey).
> Both approaches are valid and used in the real world. Ruby uses the inclusive
> ("right-closed" in math speak) first approach:
> >> l = [1,2,3,4]
> >> l.slice(1,2)
> => [2, 3]
> Python uses the exclusive ("right-open" in math speak) second approach:
> >>> l = [1,2,3,4]
> >>> l[1,2]
> [2]
> For array indices both work fine and which one to prefer is mostly an issue
> of habit. In spoken language both approaches are used: "Have the Software
> done until saturday" probably means right-open to the client and right-closed
> to the coder.
> But if you are working with keys that are more than array indexes, then
> right-open is much easier to handle. That is because you have to *guess* the
> biggest value you want to get. The Wiki at
> http://wiki.apache.org/couchdb/View_collation contains an example of that
> problem:
> It is suggested that you use
> startkey="_design/"&endkey="_design/Z"
> or
> startkey="_design/"&endkey="_design/\u″
> to get a list of all design documents - also the replication system in the db
> core uses the same hack.
> This breaks if a design document is named "ZTop" or
> "\Iñtërnâtiônàlizætiøn". Such names might be unlikely but we are computer
> scientists; "unlikely" is a bad approach to software engineering.
> The think what we really want to ask CouchDB is to "get all documents with
> keys starting with '_design/'".
> This is basically impossible to do with right-closed intervals. We could use
> startkey="_design/"&endkey="_design0″ ('0′ is the ASCII character after '/')
> and this will work fine ... until there is actually a document with the key
> "_design0″ in the system. Unlikely, but ...
> To make selection by intervals reliable currently clients have to guess the
> last key (the approach) or use the fist key not to include (the _design0
> approach) and then post process the result to remove the last element
> returned if it exactly matches the given endkey value.
> If couchdb would change to a right-open interval approach post processing
> would go away in most cases. See
> http://blogs.23.nu/c0re/2008/12/building-a-track-and-trace-application-with-couchdb/
> for two real world examples.
> At least for string keys and float keys changing the meaning to [startkey,
> endkey[ would allow selections like
> * "all strings starting with 'abc'"
> * all numbers between 10.5 and 11
> It also would hopefully break not to much existing code. Since the notion of
> endkey seems to be already considered "fishy" (see the Z approach) most
> code seems to try to avoid that issue. For example
> 'startkey="_design/"&endkey="_design/Z"' still would work unless you
> have a design document being named exactly "Z".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-290) Include sequence number in update notifications
[ https://issues.apache.org/jira/browse/COUCHDB-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695503#action_12695503 ] eric casteleijn commented on COUCHDB-290: - That is excellent news! If there is any way we can assist with that work (testing, discussion/feedback, documenting on wiki, or coding) please let us know. If this is in the near future, I guess we can live with patching our couch instances for the duration. > Include sequence number in update notifications > --- > > Key: COUCHDB-290 > URL: https://issues.apache.org/jira/browse/COUCHDB-290 > Project: CouchDB > Issue Type: Improvement >Affects Versions: 0.9 >Reporter: Elliot Murphy >Priority: Minor > Fix For: 0.10 > > Attachments: couchdb-sequences.patch, couchdb-sequences.patch > > > Hi! There's been requests to include the sequence number when sending an > update notification. Thanks to the guidance from davisp on #couchdb on March > 13th, I've been able to put together a little patch that does just that. In > the future I'm interested in doing the same for the create notification, and > perhaps extending create/delete/update notifications to include a list of > affected doc IDs. > For now though, just this simple patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-290) Include sequence number in update notifications
[ https://issues.apache.org/jira/browse/COUCHDB-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694976#action_12694976 ] eric casteleijn commented on COUCHDB-290: - Now that 0.9 has landed, is there any chance this patch could land? I found an issue in our specific use case which make it more that just optimization: we log sequence numbers with timestamps to a file, and getting the sequence number from couch with a request from the update trigger frequently returns a higher sequence number than one would expect, since another update has already happened before the update trigger gets back the sequence number. This can be coded around, but it makes it *very* hard to write deterministic tests for the update triggers. If there is anything I or Elliot can do to help this patch along, we'd be glad to do it. > Include sequence number in update notifications > --- > > Key: COUCHDB-290 > URL: https://issues.apache.org/jira/browse/COUCHDB-290 > Project: CouchDB > Issue Type: Improvement >Affects Versions: 0.9 >Reporter: Elliot Murphy >Priority: Minor > Fix For: 0.10 > > Attachments: couchdb-sequences.patch, couchdb-sequences.patch > > > Hi! There's been requests to include the sequence number when sending an > update notification. Thanks to the guidance from davisp on #couchdb on March > 13th, I've been able to put together a little patch that does just that. In > the future I'm interested in doing the same for the create notification, and > perhaps extending create/delete/update notifications to include a list of > affected doc IDs. > For now though, just this simple patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-290) Include sequence number in update notifications
[ https://issues.apache.org/jira/browse/COUCHDB-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682313#action_12682313 ] eric casteleijn commented on COUCHDB-290: - @Jan ah, yes: remembering the last sequence number you got in your last query would solve the problem for some use cases, unfortunately not for ours: we want to track sequence numbers and log them with a timestamp, i.e. we don't want to query the couchdb at all at that particular moment, but possibly at some later date for everything updated between seq_n and seq_m, based on their timestamps. > Include sequence number in update notifications > --- > > Key: COUCHDB-290 > URL: https://issues.apache.org/jira/browse/COUCHDB-290 > Project: CouchDB > Issue Type: Improvement >Affects Versions: 0.9 >Reporter: Elliot Murphy >Priority: Minor > Fix For: 0.9 > > Attachments: couchdb-sequences.patch, couchdb-sequences.patch > > > Hi! There's been requests to include the sequence number when sending an > update notification. Thanks to the guidance from davisp on #couchdb on March > 13th, I've been able to put together a little patch that does just that. In > the future I'm interested in doing the same for the create notification, and > perhaps extending create/delete/update notifications to include a list of > affected doc IDs. > For now though, just this simple patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-290) Include sequence number in update notifications
[ https://issues.apache.org/jira/browse/COUCHDB-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682301#action_12682301 ] eric casteleijn commented on COUCHDB-290: - The problem it solves is that for many update triggers, it is important to find out what has changed in the update, which unfortunately means logging sequence numbers outside of couchdb, so that an all_documents_by_seq query?startkey=old_seq can be run to find out which documents have changed. I think this is a pretty common use case. We can of course let each update trigger call fire off a request to the db to get its current seq_no, but that's unfortunate since the information could easily be provided to the update trigger, and by the time the request has returned, the sequence number might have changed again. A list of changed document ids would be even better, but if this is not possible, we will have to do requests for them. Bulk updates aside, this would always be a single document id though, or am I wrong? If there is a way to pass that document id *or* "_bulk" or something, that would solve it for all non-nulk updates at least. I realize that would probably be a more involved patch. > Include sequence number in update notifications > --- > > Key: COUCHDB-290 > URL: https://issues.apache.org/jira/browse/COUCHDB-290 > Project: CouchDB > Issue Type: Improvement >Affects Versions: 0.9 >Reporter: Elliot Murphy >Priority: Minor > Fix For: 0.9 > > Attachments: couchdb-sequences.patch, couchdb-sequences.patch > > > Hi! There's been requests to include the sequence number when sending an > update notification. Thanks to the guidance from davisp on #couchdb on March > 13th, I've been able to put together a little patch that does just that. In > the future I'm interested in doing the same for the create notification, and > perhaps extending create/delete/update notifications to include a list of > affected doc IDs. > For now though, just this simple patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-284) Couchdb can't find main.js when running under a path that contains a directory name starting with two or more dashes
Couchdb can't find main.js when running under a path that contains a directory name starting with two or more dashes Key: COUCHDB-284 URL: https://issues.apache.org/jira/browse/COUCHDB-284 Project: CouchDB Issue Type: Bug Components: Build System, Infrastructure Affects Versions: 0.9 Environment: Ubuntu Linux Reporter: eric casteleijn Priority: Minor When coucdb runs in directory below any directory whose name starts with two or more dashes, say /some/path/---foo/couch It will error when it tries to call main.js with: "could not open script file /path/some/-foo/couch/share/server/main.js" Note that the first two dashes have been dropped. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-274) Add a -C option to load configuration file without resetting system default
[ https://issues.apache.org/jira/browse/COUCHDB-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn updated COUCHDB-274: Attachment: couchdb.patch Restored ':' option to -c option. Thanks Noah. > Add a -C option to load configuration file without resetting system default > --- > > Key: COUCHDB-274 > URL: https://issues.apache.org/jira/browse/COUCHDB-274 > Project: CouchDB > Issue Type: New Feature > Components: Infrastructure >Reporter: eric casteleijn >Priority: Minor > Attachments: couchdb.patch > > > I created a patch that adds a -C (as opposed to lowercase -c) option that > accepts one of more configuration files and overrides the system default with > the values specified therein. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-274) Add a -C option to load configuration file without resetting system default
[ https://issues.apache.org/jira/browse/COUCHDB-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn updated COUCHDB-274: Attachment: (was: couchdb.patch) > Add a -C option to load configuration file without resetting system default > --- > > Key: COUCHDB-274 > URL: https://issues.apache.org/jira/browse/COUCHDB-274 > Project: CouchDB > Issue Type: New Feature > Components: Infrastructure > Reporter: eric casteleijn >Priority: Minor > > I created a patch that adds a -C (as opposed to lowercase -c) option that > accepts one of more configuration files and overrides the system default with > the values specified therein. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-274) Add a -C option to load configuration file without resetting system default
[ https://issues.apache.org/jira/browse/COUCHDB-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eric casteleijn updated COUCHDB-274: Attachment: couchdb.patch > Add a -C option to load configuration file without resetting system default > --- > > Key: COUCHDB-274 > URL: https://issues.apache.org/jira/browse/COUCHDB-274 > Project: CouchDB > Issue Type: New Feature > Components: Infrastructure > Reporter: eric casteleijn >Priority: Minor > Attachments: couchdb.patch > > > I created a patch that adds a -C (as opposed to lowercase -c) option that > accepts one of more configuration files and overrides the system default with > the values specified therein. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-274) Add a -C option to load configuration file without resetting system default
Add a -C option to load configuration file without resetting system default --- Key: COUCHDB-274 URL: https://issues.apache.org/jira/browse/COUCHDB-274 Project: CouchDB Issue Type: New Feature Components: Infrastructure Reporter: eric casteleijn Priority: Minor I created a patch that adds a -C (as opposed to lowercase -c) option that accepts one of more configuration files and overrides the system default with the values specified therein. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Proposal: let -c *not* ignore default.ini
Noah Slater wrote: On Fri, Feb 27, 2009 at 07:48:17PM +0100, eric casteleijn wrote: For me, this weird behaviour breaks the rule of least surprise. I understand that, but for me it was actually the reverse. I expected the default.ini to be used unless I specified an option to not use it, and anything I passed into -c to customize the settings I wanted to override. Perhaps it is my zc.buildout history, where you only ever state what needs to be changed, and can even do things like: The --help output is very clear about this: -c FILE use configuration FILE (chainable, resets system default) I can see that a consensus may not be within reach here, so perhaps another option that does provide this behavior could be added, and if we can find clear enough descriptions to distinguish the two. Perhaps override vs. custom(ize) although all the obvious letters have already been taken... I would accept a patch which added: -c FILE use configuration FILE (chainable, reset system default) -C FILE use configuration FILE (chainable, do not reset system default) That seems like the best option, I shall write a patch, and put it in JIRA. Thanks for the feedback, all. -- - eric casteleijn http://www.canonical.com
Re: Proposal: let -c *not* ignore default.ini
Noah Slater wrote: On Fri, Feb 27, 2009 at 02:17:38PM +0100, eric casteleijn wrote: The only potential problem I see is if the current way .ini files are handled does not allow you to *un*specify a particular option, in which case you'd need to have an option to ignore the default.ini. We cannot accept this patch as is, because it provides no way to avoid the default configuration file. As it currently stands, if you wish to use some custom configuration file, it assumed that you know enough about what you're doing to add in the default or local configuration files as well. To accept your patch, we would need an additional option, like you mention: --ignore-default-configuration For me, this weird behaviour breaks the rule of least surprise. I understand that, but for me it was actually the reverse. I expected the default.ini to be used unless I specified an option to not use it, and anything I passed into -c to customize the settings I wanted to override. Perhaps it is my zc.buildout history, where you only ever state what needs to be changed, and can even do things like: multivalued_option+=value multivalued_option-=value which makes for an extremely flexible configuration system. Even with the modification suggested, I doubt I would want to accept this patch. I suggest that you use the following on your local system: couchdb -c default.ini -c custom.ini That is possible, but somewhat undesirable in our setup, since I'm starting several couch servers, and the default.ini and the custom ones will be in different paths, which I would have to compute and pass into that script. Again, not impossible, but I would prefer an option not to do that. I can see that a consensus may not be within reach here, so perhaps another option that does provide this behavior could be added, and if we can find clear enough descriptions to distinguish the two. Perhaps override vs. custom(ize) although all the obvious letters have already been taken... -- - eric casteleijn http://www.canonical.com
Proposal: let -c *not* ignore default.ini
Now that local.ini overrides the default.ini, I would expect the default.ini not to be ignored but to be overridden by whatever is passed into couchdb with the -c option, so you only need to specify that which you want to change, even when you use a local.ini on a specified path. (patch attached to do this, but against couchdb, should probably be against couchdb.tpl and/or couchdb.tpl.in but my make-fu isn't all that, coming from Python.) The only potential problem I see is if the current way .ini files are handled does not allow you to *un*specify a particular option, in which case you'd need to have an option to ignore the default.ini. If my proposal is not acceptable, perhaps I can rewrite it to an additional command line option, that does work in this way. -- - eric casteleijn http://www.canonical.com --- couchdb 2009-02-27 12:20:37.0 +0100 +++ couchdbnew 2009-02-27 13:45:36.0 +0100 @@ -168,7 +168,7 @@ interactive_option="+Bd -noinput" fi if test -n "$INI_FILES"; then -ini_files="$INI_FILES" +ini_files="$DEFAULT_INI_FILE $INI_FILES" else ini_files="$DEFAULT_INI_FILE $LOCAL_INI_FILE" fi
