[jira] Updated: (COUCHDB-272) timeout on DELETE
[
https://issues.apache.org/jira/browse/COUCHDB-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Hinrichs updated COUCHDB-272:
--
Attachment: couchdb_python62_Test.py
script shows the error occurring on my system. depending on disk subsystem
speed the size of the db necessary will vary.
If you comment out the line #63
del srvA[dbAname]
you can attempt the delete with curl and see the same results
curl -X DELETE http://localhost:5984/couchdb-python-62
> timeout on DELETE
> -
>
> Key: COUCHDB-272
> URL: https://issues.apache.org/jira/browse/COUCHDB-272
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.9
> Environment: apache couchdb 0.9.0a748865
>Reporter: Jeff Hinrichs
> Attachments: couchdb_python62_Test.py
>
>
> issuing a DELETE on a database of moderate size(4GB or larger) results in
> couchdb returning a timeout error.
> (demo-couchdb)j...@jlh-d520:~/Projects/venvs/demo-couchdb/src$ curl -X DELETE
> http://localhost:5984/couchdb-python-62
> {"error":"timeout","reason":"{gen_server,call,\n[couch_server,\n
>{delete,<<\"couchdb-python-62\">>,\n
> [{user_ctx,{user_ctx,null,[<<\"_admin\">>]}}]}]}"}
> The database is deleted however the calling code is given bad information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-272) timeout on DELETE
timeout on DELETE
-
Key: COUCHDB-272
URL: https://issues.apache.org/jira/browse/COUCHDB-272
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 0.9
Environment: apache couchdb 0.9.0a748865
Reporter: Jeff Hinrichs
issuing a DELETE on a database of moderate size(4GB or larger) results in
couchdb returning a timeout error.
(demo-couchdb)j...@jlh-d520:~/Projects/venvs/demo-couchdb/src$ curl -X DELETE
http://localhost:5984/couchdb-python-62
{"error":"timeout","reason":"{gen_server,call,\n[couch_server,\n
{delete,<<\"couchdb-python-62\">>,\n
[{user_ctx,{user_ctx,null,[<<\"_admin\">>]}}]}]}"}
The database is deleted however the calling code is given bad information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-270) Replication w/ Large Attachments Fails
[
https://issues.apache.org/jira/browse/COUCHDB-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Hinrichs updated COUCHDB-270:
--
Attachment: couchdb270_Test.py
The attached script can produce a number of errors in replication. Hopefully a
python based script is helpful for you. The script requires couchdb-python 0.5
and nose (to run the tests)
One thing I have discovered is that the replication issues are not limited to
attachments but are related to overall document size. I am creating these
replication issues by only using docs of large size.
You will need to modify the top of the script
srvAuri = 'http://192.168.2.52:5984/'
srvBuri = 'http://192.168.2.194:5984/'
to make sense for your environment
The bigger the document size the harder/faster couchdb dies. 1MB documents are
enough to make it groan before going away, 10MB will occasionally garner a
wimpering before death -- though not always, while 20MB documents are akin to a
head shot.
> Replication w/ Large Attachments Fails
> --
>
> Key: COUCHDB-270
> URL: https://issues.apache.org/jira/browse/COUCHDB-270
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.9
> Environment: Apache CouchDB 0.9.0a748379
>Reporter: Jeff Hinrichs
> Attachments: couchdb270_Test.py
>
>
> Attempting to replicate a database with largish attachments (<= ~18MB of
> attachments in a doc, less thatn 200 docs) from one machine to another fails
> consistently and at the same point.
> Scenario:
> Both servers are running from HEAD and I've been tracking for some time.
> This problem has been around as long as I've been using couch.
> Machine A holds the original database, Machine B is the server that is doing
> a PULL replication
> During the replication, Machine A starts showing the following sporadically
> in the log:
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5902.3>] 'GET'
> /delasco-invoices/INV00652429?revs=true&attachments=true&latest=true&open_revs=["425644723"]
> {1,
> 1}
> Headers: [{'Host',"192.168.2.52:5984"}]
> [Fri, 27 Feb 2009 14:02:48 GMT] [error] [<0.5901.3>] Uncaught error in
> HTTP request: {exit,normal}
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5901.3>] Stacktrace:
> [{mochiweb_request,send,2},
> {couch_httpd,send_chunk,2},
> {couch_httpd_db,db_doc_req,3},
> {couch_httpd_db,do_db_req,2},
> {couch_httpd,handle_request,3},
> {mochiweb_http,headers,5},
> {proc_lib,init_p,5}]
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5901.3>] HTTPd 500 error response:
> {"error":"error","reason":"normal"}
> As the replication continues, the frequency of these error "Uncaught error in
> HTTP request: {exit,normal}" increase. Until the error is being constantly
> repeated. Then Machine B stops sending requests, no more log output, no
> errors, the last thing in Machine B's log file is:
> [Fri, 27 Feb 2009 14:03:24 GMT] [info] [<0.20893.1>] retrying
> couch_rep HTTP get request due to {error, req_timedout}: [104,116,
> 116,112,58,
> 47,47,49,
> 57,50,46,
> 49,54,56,
> 46,50,46,
> 53,50,58,
> 53,57,56,
> 52,47,100,
> 101,108,97,
> 115,99,111,
> 45,105,110,
> 118,111,
> 105,99,101,
> 115,47,73,
> 78,86,48,
> 48,54,53,
> 50,49,51,
> 56,63,114,
> 101,118,
> 115,61,116,
> 114,117,
> 101,38,97,
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: list_multikey.3.diff Refactored start and row functions out to reduce code duplication. > _list doesn't allow POST requests > - > > Key: COUCHDB-269 > URL: https://issues.apache.org/jira/browse/COUCHDB-269 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Reporter: Jason Davies > Attachments: list_multikey.2.diff, list_multikey.3.diff, > list_multikey.diff > > > Currently, a 405 response is returned if you issue a POST request to _list. > We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-270) Replication w/ Large Attachments Fails
[
https://issues.apache.org/jira/browse/COUCHDB-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677694#action_12677694
]
Jeff Hinrichs commented on COUCHDB-270:
---
The above failure was created using futon.
I am trying to build a script to replicate the failure on demand but I am
seeing something different now, the error is kind of like couchdb-240 (see
below for more error output) I will attach my script to this issue. Hopefully
it will help.
--error information---
[Sat, 28 Feb 2009 16:56:35 GMT] [info] [<0.962.0>] 192.168.2.52 - - 'POST'
/couchdb-270-target/_bulk_docs 201
[Sat, 28 Feb 2009 16:56:49 GMT] [debug] [<0.979.0>] 'POST'
/couchdb-270-target/_bulk_docs {1,1}
Headers: [{'Host',"192.168.2.194:5984"},
{'Transfer-Encoding',"chunked"}]
[Sat, 28 Feb 2009 16:57:18 GMT] [error] [<0.906.0>] ** Generic server <0.906.0>
terminating
** Last message in was {update_docs,[[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[],[],[],[],[],[],[]],
[]}
** When Server state == {db,<0.905.0>,<0.906.0>,nil,<<"1235840109444771">>,
<0.904.0>,<0.908.0>,
{db_header,0,100,
{104867784,0},
{104876441,{100,0}},
{104872023,100},
nil,0,nil,nil},
{stream,<0.907.0>,<0.904.0>},
{btree,<0.904.0>,
{104876441,{100,0}},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,<0.904.0>,
{104872023,100},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,<0.904.0>,nil,
#Fun,
#Fun,
#Fun,nil},
100,<<"couchdb-270-target">>,
"/usr/local/var/lib/couchdb/couchdb-270-target.couch",
[],[],nil,
{user_ctx,null,[]},
nil}
** Reason for termination ==
** {function_clause,
[{couch_db_updater,'-update_docs_int/3-fun-0-',[[],{[],[]}]},
{lists,foldl,3},
{couch_db_updater,update_docs_int,3},
{couch_db_updater,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p,5}]}
[Sat, 28 Feb 2009 16:57:19 GMT] [error] [<0.906.0>] {error_report,<0.22.0>,
{<0.906.0>,crash_report,
[[{pid,<0.906.0>},
{registered_name,[]},
{error_info,
{exit,
{function_clause,
[{couch_db_updater,'-update_docs_int/3-fun-0-',
[[],{[],[]}]},
{lists,foldl,3},
{couch_db_updater,update_docs_int,3},
{couch_db_updater,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p,5}]},
[{gen_server,terminate,6},{proc_lib,init_p,5}]}},
{initial_call,
{gen,init_it,
[gen_server,<0.905.0>,<0.905.0>,couch_db_updater,
{<0.905.0>,<<"couchdb-270-target">>,
"/usr/local/var/lib/couchdb/couchdb-270-target.couch",
<0.904.0>,
[create,{user_ctx,{user_ctx,null,[<<"_admin">>]}}]},
[]]}},
{ancestors,
[<0.905.0>,couch_server,couch_primary_services,
couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.905.0>,<0.907.0>]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,6765},
{stack_size,23},
{reductions,96260}],
[{neighbour,
[{pid,<0.907.0>},
{registered_name,[]},
{initial_call,
{gen,init_it,
[gen_server,<0.906.0>,<0.906.0>,couch_stream,
{{0,0},<0.904.0>},
[]]}},
{current_function,{gen_server,loop,6}},
{ancestors,
[<0.906.0>,<0.905.0>,couch_server,couch_prim
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Comment: was deleted > _list doesn't allow POST requests > - > > Key: COUCHDB-269 > URL: https://issues.apache.org/jira/browse/COUCHDB-269 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Reporter: Jason Davies > Attachments: list_multikey.2.diff, list_multikey.diff > > > Currently, a 405 response is returned if you issue a POST request to _list. > We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: list_multikey.2.diff Updated patch for latest trunk (r748754) after jchris's ETag update. > _list doesn't allow POST requests > - > > Key: COUCHDB-269 > URL: https://issues.apache.org/jira/browse/COUCHDB-269 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Reporter: Jason Davies > Attachments: list_multikey.2.diff, list_multikey.diff > > > Currently, a 405 response is returned if you issue a POST request to _list. > We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: (was: list_multikey.2.diff) > _list doesn't allow POST requests > - > > Key: COUCHDB-269 > URL: https://issues.apache.org/jira/browse/COUCHDB-269 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Reporter: Jason Davies > Attachments: list_multikey.2.diff, list_multikey.diff > > > Currently, a 405 response is returned if you issue a POST request to _list. > We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-269) _list doesn't allow POST requests
[ https://issues.apache.org/jira/browse/COUCHDB-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Davies updated COUCHDB-269: - Attachment: list_multikey.2.diff Updated patch for latest trunk (r748754) after jchris's ETag update. > _list doesn't allow POST requests > - > > Key: COUCHDB-269 > URL: https://issues.apache.org/jira/browse/COUCHDB-269 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Reporter: Jason Davies > Attachments: list_multikey.2.diff, list_multikey.diff > > > Currently, a 405 response is returned if you issue a POST request to _list. > We should allow POST requests so we can do a multi-key fetch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677669#action_12677669 ] Louis Gerbarg commented on COUCHDB-271: --- On darwin you do this by setting the F_NOCACHE fcntl on the descriptor: #include err = fcntl(file_des, F_NOCACHE); > preventing compaction from ruining the OS block cache > - > > Key: COUCHDB-271 > URL: https://issues.apache.org/jira/browse/COUCHDB-271 > Project: CouchDB > Issue Type: Improvement > Components: Database Core >Affects Versions: 0.8.1, 0.9 >Reporter: Jan Lehnardt > > Adam Kocolosk: > Hi, I've noticed that compacting large DBs pretty much kills any filesystem > caching benefits for CouchDB. I believe the problem is that the OS (Linux > 2.6.21 kernel in my case) is caching blocks from the .compact file, even > though those blocks won't be read again until compaction has finished. In > the meantime, the portion of the cache dedicated to the old DB file shrinks > and performance really suffers. > I think a better mode of operation would be to advise/instruct the OS not to > cache any portion of the .compact file until we're ready to replace the main > DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() > seems like the way to go: > http://linux.die.net/man/2/posix_fadvise > This link has a little more detail and a usage example: > http://insights.oetiker.ch/linux/fadvise.html > Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang > VM. Perhaps the simplest approach would be to have a helper process that we > can spawn which calls that function (or its equivalent on a non-Linux OS) > periodically during compaction? I'm not really sure, but I wanted to get > this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: REST, Hypermedia, and CouchApps
On 27 Feb 2009, at 19:04, Chris Anderson wrote: This also seems sensible to me. However, perhaps the name "_design" is no longer meaningful. The namespace would no longer cover just the definition of the map/reduce and list/show functionality, but also all of its output. The name _design is arbitrary. It is not in any way connected to "views" in a meaningful way. Except by definition. Damien envisioned, and the technical outline hints at this, that a design document includes all definitions and resources for a single application (yes, CouchApps is just a consequence of CouchDB's design :). They are meant as "design" documents for applications, but not exclusively, see below. When I first came to the realization that to make relative links work, the urls would have to get longer, I thought "hey, let's change _design to _app" I didn't bring it up in the first round because I didn't want to muddy the waters. But now that it's brought up... Hey, let's change _design/ to _app/ _app suggests that you are writing some kind of app in a _app/_design document. It make less sense in the case where CouchDB is used as a massive database backend. Views are likely split between multiple design docs because of the evaluation behaviour. I think _design is just fine. To make a non-intrusive change for CouchApps, we could alias _app to _design for the nicer URLs. But then, this is probably cruft that we might want to avoid early on. (*) Such as rule would also allow other potential future uses, e.g. /db/docid/_plugin that is kind of a neat consequence of a no _rule for attachments. The other alternative is to keep attachments in a dedicated namespace /db/docid/_attachments/index.html The only thing I don't like about this is that /db/docid/index.html is kinda neat. I'm not a fan of Key-Value-URLing where it can be avoided. I'd say the no _ rule is worth enforcing (and consistent with other uses of _). Cheers Jan --
[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677667#action_12677667 ] Jan Lehnardt commented on COUCHDB-271: -- The best way to get any patch into OTP is coming up with a patch to send to erlang-patc...@. If we manage to find cross-platform alternatives and a non-intrusive patch, I'd assume a high chance of Ericsson accepting the patch. We could make it easy and label the patch Linux-only, but we might as well do some research for at least Windows, Solaris, the BSD's and Darwin. Once included and released with OTP, we'd bump the minimum required version. If either the patch gets rejected or want to support older OTP releases, we should look into the external daemon variant. > preventing compaction from ruining the OS block cache > - > > Key: COUCHDB-271 > URL: https://issues.apache.org/jira/browse/COUCHDB-271 > Project: CouchDB > Issue Type: Improvement > Components: Database Core >Affects Versions: 0.8.1, 0.9 >Reporter: Jan Lehnardt > > Adam Kocolosk: > Hi, I've noticed that compacting large DBs pretty much kills any filesystem > caching benefits for CouchDB. I believe the problem is that the OS (Linux > 2.6.21 kernel in my case) is caching blocks from the .compact file, even > though those blocks won't be read again until compaction has finished. In > the meantime, the portion of the cache dedicated to the old DB file shrinks > and performance really suffers. > I think a better mode of operation would be to advise/instruct the OS not to > cache any portion of the .compact file until we're ready to replace the main > DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() > seems like the way to go: > http://linux.die.net/man/2/posix_fadvise > This link has a little more detail and a usage example: > http://insights.oetiker.ch/linux/fadvise.html > Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang > VM. Perhaps the simplest approach would be to have a helper process that we > can spawn which calls that function (or its equivalent on a non-Linux OS) > periodically during compaction? I'm not really sure, but I wanted to get > this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: preventing compaction from ruining the OS block cache
moved this to https://issues.apache.org/jira/browse/COUCHDB-271 so it doesn't fall between the cracks. Cheers Jan -- On 27 Feb 2009, at 19:32, Damien Katz wrote: Oh yeah, one more option that is kind of crazy is to spawn a small external child process for file io. It would be a very small simple process that opens a file and responds to read/write commands from the erlang server. Then we can implement exactly the low level apis and caching behavior desired. The cost is extra IPC, but that should be small compare the the cost of a blown file cache. -Damien On Feb 27, 2009, at 1:23 PM, Damien Katz wrote: The problem is we don't get access to the low level apis or flags passed in to the OS unless Erlang chooses to expose it. We have similar problems with compaction on windows because we need special flags to give us unix file semantics. To fix this, we'll either need the Erlang VM changed or use our own Erlang file driver interface. -Damien On Feb 26, 2009, at 8:25 PM, Adam Kocoloski wrote: Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB. I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished. In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers. I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go: http://linux.die.net/man/2/posix_fadvise This link has a little more detail and a usage example: http://insights.oetiker.ch/linux/fadvise.html Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM. Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction? I'm not really sure, but I wanted to get this out on the list for discussion. Best, Adam
[jira] Commented: (COUCHDB-271) preventing compaction from ruining the OS block cache
[ https://issues.apache.org/jira/browse/COUCHDB-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677666#action_12677666 ] Jan Lehnardt commented on COUCHDB-271: -- Damien Katz: The problem is we don't get access to the low level apis or flags passed in to the OS unless Erlang chooses to expose it. We have similar problems with compaction on windows because we need special flags to give us unix file semantics. To fix this, we'll either need the Erlang VM changed or use our own Erlang file driver interface. -- Oh yeah, one more option that is kind of crazy is to spawn a small external child process for file io. It would be a very small simple process that opens a file and responds to read/write commands from the erlang server. Then we can implement exactly the low level apis and caching behavior desired. The cost is extra IPC, but that should be small compare the the cost of a blown file cache. > preventing compaction from ruining the OS block cache > - > > Key: COUCHDB-271 > URL: https://issues.apache.org/jira/browse/COUCHDB-271 > Project: CouchDB > Issue Type: Improvement > Components: Database Core >Affects Versions: 0.8.1, 0.9 >Reporter: Jan Lehnardt > > Adam Kocolosk: > Hi, I've noticed that compacting large DBs pretty much kills any filesystem > caching benefits for CouchDB. I believe the problem is that the OS (Linux > 2.6.21 kernel in my case) is caching blocks from the .compact file, even > though those blocks won't be read again until compaction has finished. In > the meantime, the portion of the cache dedicated to the old DB file shrinks > and performance really suffers. > I think a better mode of operation would be to advise/instruct the OS not to > cache any portion of the .compact file until we're ready to replace the main > DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() > seems like the way to go: > http://linux.die.net/man/2/posix_fadvise > This link has a little more detail and a usage example: > http://insights.oetiker.ch/linux/fadvise.html > Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang > VM. Perhaps the simplest approach would be to have a helper process that we > can spawn which calls that function (or its equivalent on a non-Linux OS) > periodically during compaction? I'm not really sure, but I wanted to get > this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-271) preventing compaction from ruining the OS block cache
preventing compaction from ruining the OS block cache - Key: COUCHDB-271 URL: https://issues.apache.org/jira/browse/COUCHDB-271 Project: CouchDB Issue Type: Improvement Components: Database Core Affects Versions: 0.8.1, 0.9 Reporter: Jan Lehnardt Adam Kocolosk: Hi, I've noticed that compacting large DBs pretty much kills any filesystem caching benefits for CouchDB. I believe the problem is that the OS (Linux 2.6.21 kernel in my case) is caching blocks from the .compact file, even though those blocks won't be read again until compaction has finished. In the meantime, the portion of the cache dedicated to the old DB file shrinks and performance really suffers. I think a better mode of operation would be to advise/instruct the OS not to cache any portion of the .compact file until we're ready to replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to go: http://linux.die.net/man/2/posix_fadvise This link has a little more detail and a usage example: http://insights.oetiker.ch/linux/fadvise.html Of course, POSIX_FADV_DONTNEED isn't really available from inside the Erlang VM. Perhaps the simplest approach would be to have a helper process that we can spawn which calls that function (or its equivalent on a non-Linux OS) periodically during compaction? I'm not really sure, but I wanted to get this out on the list for discussion. Best, -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
