[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12677666#action_12677666 ] Jan Lehnardt commented on COUCHDB-271: -- Damien Katz: The problem is we don't get access to the low level apis or flags passed in to the OS unless Erlang chooses to expose it. We have similar problems with compaction on windows because we need special flags to give us unix file semantics. To fix this, we'll either need the Erlang VM changed or use our own Erlang file driver interface. -- Oh yeah, one more option that is kind of crazy is to spawn a small external child process for file io. It would be a very small simple process that opens a file and responds to read/write commands from the erlang server. Then we can implement exactly the low level apis and caching behavior desired. The cost is extra IPC, but that should be small compare the the cost of a blown file cache. preventing compaction from ruining the OS block cache - Key: COUCHDB-271 URL: https://issues.apache.org/jira/browse/COUCHDB-271 Project: CouchDB Issue Type: Improvement Components: Database Core Affects Versions: 0.8.1, 0.9 Reporter: Jan Lehnardt Adam Kocolosk: Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB. I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished. In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers. I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go: http://linux.die.net/man/2/posix_fadvise This link has a little more detail and a usage example: http://insights.oetiker.ch/linux/fadvise.html Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM. Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction? I'm not really sure, but I wanted to get this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12677667#action_12677667 ] Jan Lehnardt commented on COUCHDB-271: -- The best way to get any patch into OTP is coming up with a patch to send to erlang-patc...@. If we manage to find cross-platform alternatives and a non-intrusive patch, I'd assume a high chance of Ericsson accepting the patch. We could make it easy and label the patch Linux-only, but we might as well do some research for at least Windows, Solaris, the BSD's and Darwin. Once included and released with OTP, we'd bump the minimum required version. If either the patch gets rejected or want to support older OTP releases, we should look into the external daemon variant. preventing compaction from ruining the OS block cache - Key: COUCHDB-271 URL: https://issues.apache.org/jira/browse/COUCHDB-271 Project: CouchDB Issue Type: Improvement Components: Database Core Affects Versions: 0.8.1, 0.9 Reporter: Jan Lehnardt Adam Kocolosk: Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB. I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished. In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers. I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go: http://linux.die.net/man/2/posix_fadvise This link has a little more detail and a usage example: http://insights.oetiker.ch/linux/fadvise.html Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM. Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction? I'm not really sure, but I wanted to get this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: REST, Hypermedia, and CouchApps
On 27 Feb 2009, at 19:04, Chris Anderson wrote: This also seems sensible to me. However, perhaps the name _design is no longer meaningful. The namespace would no longer cover just the definition of the map/reduce and list/show functionality, but also all of its output. The name _design is arbitrary. It is not in any way connected to views in a meaningful way. Except by definition. Damien envisioned, and the technical outline hints at this, that a design document includes all definitions and resources for a single application (yes, CouchApps is just a consequence of CouchDB's design :). They are meant as design documents for applications, but not exclusively, see below. When I first came to the realization that to make relative links work, the urls would have to get longer, I thought hey, let's change _design to _app I didn't bring it up in the first round because I didn't want to muddy the waters. But now that it's brought up... Hey, let's change _design/ to _app/ _app suggests that you are writing some kind of app in a _app/_design document. It make less sense in the case where CouchDB is used as a massive database backend. Views are likely split between multiple design docs because of the evaluation behaviour. I think _design is just fine. To make a non-intrusive change for CouchApps, we could alias _app to _design for the nicer URLs. But then, this is probably cruft that we might want to avoid early on. (*) Such as rule would also allow other potential future uses, e.g. /db/docid/_plugin that is kind of a neat consequence of a no _rule for attachments. The other alternative is to keep attachments in a dedicated namespace /db/docid/_attachments/index.html The only thing I don't like about this is that /db/docid/index.html is kinda neat. I'm not a fan of Key-Value-URLing where it can be avoided. I'd say the no _ rule is worth enforcing (and consistent with other uses of _). Cheers Jan --
[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12677669#action_12677669 ] Louis Gerbarg commented on COUCHDB-271: --- On darwin you do this by setting the F_NOCACHE fcntl on the descriptor: #include sys/fcntl.h err = fcntl(file_des, F_NOCACHE); preventing compaction from ruining the OS block cache - Key: COUCHDB-271 URL: https://issues.apache.org/jira/browse/COUCHDB-271 Project: CouchDB Issue Type: Improvement Components: Database Core Affects Versions: 0.8.1, 0.9 Reporter: Jan Lehnardt Adam Kocolosk: Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB. I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished. In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers. I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go: http://linux.die.net/man/2/posix_fadvise This link has a little more detail and a usage example: http://insights.oetiker.ch/linux/fadvise.html Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM. Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction? I'm not really sure, but I wanted to get this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: (was: list_multikey.2.diff) _list doesn't allow POST requests - Key: COUCHDB-269 URL: https://issues.apache.org/jira/browse/COUCHDB-269 Project: CouchDB Issue Type: Bug Components: HTTP Interface Reporter: Jason Davies Attachments: list_multikey.2.diff, list_multikey.diff Currently, a 405 response is returned if you issue a POST request to _list. We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: list_multikey.3.diff Refactored start and row functions out to reduce code duplication. _list doesn't allow POST requests - Key: COUCHDB-269 URL: https://issues.apache.org/jira/browse/COUCHDB-269 Project: CouchDB Issue Type: Bug Components: HTTP Interface Reporter: Jason Davies Attachments: list_multikey.2.diff, list_multikey.3.diff, list_multikey.diff Currently, a 405 response is returned if you issue a POST request to _list. We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-270) Replication w/ Large Attachments Fails
[
https://issues.apache.org/jira/browse/COUCHDB-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Hinrichs updated COUCHDB-270:
--
Attachment: couchdb270_Test.py
The attached script can produce a number of errors in replication. Hopefully a
python based script is helpful for you. The script requires couchdb-python 0.5
and nose (to run the tests)
One thing I have discovered is that the replication issues are not limited to
attachments but are related to overall document size. I am creating these
replication issues by only using docs of large size.
You will need to modify the top of the script
srvAuri = 'http://192.168.2.52:5984/'
srvBuri = 'http://192.168.2.194:5984/'
to make sense for your environment
The bigger the document size the harder/faster couchdb dies. 1MB documents are
enough to make it groan before going away, 10MB will occasionally garner a
wimpering before death -- though not always, while 20MB documents are akin to a
head shot.
Replication w/ Large Attachments Fails
--
Key: COUCHDB-270
URL: https://issues.apache.org/jira/browse/COUCHDB-270
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 0.9
Environment: Apache CouchDB 0.9.0a748379
Reporter: Jeff Hinrichs
Attachments: couchdb270_Test.py
Attempting to replicate a database with largish attachments (= ~18MB of
attachments in a doc, less thatn 200 docs) from one machine to another fails
consistently and at the same point.
Scenario:
Both servers are running from HEAD and I've been tracking for some time.
This problem has been around as long as I've been using couch.
Machine A holds the original database, Machine B is the server that is doing
a PULL replication
During the replication, Machine A starts showing the following sporadically
in the log:
[Fri, 27 Feb 2009 14:02:48 GMT] [debug] [0.5902.3] 'GET'
/delasco-invoices/INV00652429?revs=trueattachments=truelatest=trueopen_revs=[425644723]
{1,
1}
Headers: [{'Host',192.168.2.52:5984}]
[Fri, 27 Feb 2009 14:02:48 GMT] [error] [0.5901.3] Uncaught error in
HTTP request: {exit,normal}
[Fri, 27 Feb 2009 14:02:48 GMT] [debug] [0.5901.3] Stacktrace:
[{mochiweb_request,send,2},
{couch_httpd,send_chunk,2},
{couch_httpd_db,db_doc_req,3},
{couch_httpd_db,do_db_req,2},
{couch_httpd,handle_request,3},
{mochiweb_http,headers,5},
{proc_lib,init_p,5}]
[Fri, 27 Feb 2009 14:02:48 GMT] [debug] [0.5901.3] HTTPd 500 error response:
{error:error,reason:normal}
As the replication continues, the frequency of these error Uncaught error in
HTTP request: {exit,normal} increase. Until the error is being constantly
repeated. Then Machine B stops sending requests, no more log output, no
errors, the last thing in Machine B's log file is:
[Fri, 27 Feb 2009 14:03:24 GMT] [info] [0.20893.1] retrying
couch_rep HTTP get request due to {error, req_timedout}: [104,116,
116,112,58,
47,47,49,
57,50,46,
49,54,56,
46,50,46,
53,50,58,
53,57,56,
52,47,100,
101,108,97,
115,99,111,
45,105,110,
118,111,
105,99,101,
115,47,73,
78,86,48,
48,54,53,
50,49,51,
56,63,114,
101,118,
115,61,116,
114,117,
101,38,97,
116,116,97,
