Re: Windows 0.11 snapshot
On 26 Feb 2010, at 14:42, Mark Hammond wrote: > On 27/02/2010 12:31 AM, Noah Slater wrote: >> I'm cutting the release soon. Are you confident that I can do that with your >> artefacts? >> >> What day do you want to hand them over to me for the official vote? > > I thought we decided that the artefacts needed to be built by me from your > signed copy of the tarball? I admit I have never tried to build from such a > tarball (just an svn tree), so maybe I need to verify that first. Are there > any such 0.11 tarballs I could use for a practice run? http://build.couchdb.org/?C=M;O=D (or just run make distcheck in your svn checkout) Cheers Jan --
Re: Windows 0.11 snapshot
On Sat, Feb 27, 2010 at 1:00 AM, Mark Hammond wrote: > Regarding the file versioning - I'm sorry that we haven't kept you in the > loop a little better, but with Damien's guidance we think we have found a > better strategy for this. nb, it's great that the issue gets attention. > > The root of this strategy comes from a realization that if a file is opened > on windows with FILE_SHARE_DELETE, the file can be deleted *or renamed* > while it is open. One limitation is that a file of the same name can not be > re-created while the old one still has handles open (the 'deleted but still > open' file still appears in directory listings until the handle is closed, > for example) > > Given this, what we can do is something like: > > * Arrange for erlang to be able to open the DB and view files with this > flag. > * Instead of deleting a file before replacing it, we first rename the file > to a unique name (ie, based on a UUID) in a special directory. > * As couch starts up, attempt to delete any old files in this special > directory. In theory, no such files should exist - the OS should take care > of actually removing any such files even if erlang crashes. > > The end result of this is that things can be made to work with a lot less > friction than the 'file versioning' scheme. I've a patch to couchdb that > works when used with a patch to erlang to open *all* files with that flag. > The next step down this path seems to be to create an erlang 'port driver' > for disk IO and use this instead of the builtin erlang file objects. We've > identified a 'bfile' erlang port driver we may be able to fork and adapt to > our needs - using our own port driver would also allow optimizations to the > file IO for non-windows platforms (eg, indicate to the OS that certain files > should not get OS-level caching, etc) > Sounds like a good plan. I noticed Jan just filed a proposal to erlang-patches for adding FILE_SHARE_DELETE flag to CreateFile in erts. > Unfortunately, none of this has been discussed anywhere formal - much of it > was on IRC - so I apologize if this is the first you heard of it. But > whatever strategy we wind up with for this is almost certainly not going to > land and be tested enough to make 0.11. Any idea when it'll land to trunk? I could use a rough estimate to decide if I should temporarily deploy the file versioning patch. cheers, -juhani
[jira] Closed: (COUCHDB-597) Replication tasks crash.
[
https://issues.apache.org/jira/browse/COUCHDB-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Lehnardt closed COUCHDB-597.
Resolution: Fixed
Fix Version/s: 0.11
Applied in trunk in r916868 and 0.11.x in r916869.
> Replication tasks crash.
>
>
> Key: COUCHDB-597
> URL: https://issues.apache.org/jira/browse/COUCHDB-597
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.11
>Reporter: Robert Newson
> Fix For: 0.11
>
> Attachments:
> 0001-changes-replication-timeouts-and-att.-fixes-COUCHDB-.patch,
> couchdb_597.patch
>
>
> If I kick off 10 replication tasks in quick succession, occasionally one or
> two of the replication tasks will die and not be resumed. It seems that the
> stat tracking is a little buggy, and under stress can eventually cause a
> permanent failure of the supervised replication task;
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [<0.80.0>] {error_report,<0.30.0>,
> {<0.80.0>,supervisor_report,
> [{supervisor,{local,couch_rep_sup}},
> {errorContext,shutdown_error},
> {reason,killed},
> {offender,
> [{pid,<0.6700.11>},
>{name,"fcbb13200a1618cf983b347f4d2c9835+create_target"},
>{mfa,
>{gen_server,start_link,
>[couch_rep,
> ["fcbb13200a1618cf983b347f4d2c9835",
> {[{<<"create_target">>,true},
>{<<"source">>,<<"http://node:5984/perf-p2";>>},
>{<<"target">>,<<"perf-p2">>}]},
> {user_ctx,null,[<<"_admin">>]}],
> []]}},
>{restart_type,temporary},
>{shutdown,1},
>{child_type,worker}]}]}}
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [emulator] Error in process
> <0.6705.11> with exit value:
> {badarg,[{ets,insert,[stats_hit_table,{{couchdb,open_os_files},-1}]},{couch_stats_collector,decrement,1}]}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: Releasing 0.11, Request for Comments
On Feb 26, 2010, at 7:00 AM, Filipe David Manana wrote: > I would still like to see ticket 639 in 0.11. > I'm reading 639 and it seems like a great patch. But it's a little bit big and I can't tell for certain that there isn't something subtle I'm not seeing. I think the best course of action would be to apply it just after the 0.11 release, so it gets some field testing before it goes out in a release. It's not a new feature so it is still a candidate for 0.11.1 / 1.0. I'd just feel better about introducing a large patch to the code base if it has time to get used in a few different settings before being baked into a release. Chris > Adam told me, via IRC, he will review the patch by the end of this week. > > I vote +1 on it. > > cheers > > On Fri, Feb 26, 2010 at 3:44 PM, Noah Slater wrote: > >> Nope, you can send payment to any email. >> >> As long as the recipient can click on the link in the email, they can >> deposit it into any account. >> >> On 26 Feb 2010, at 14:37, till wrote: >> >>> I always knew secretly open source worked like that. ;) >>> >>> Btw, I know it's not so subtle, but you need to include your paypal >> email... :D >>> >>> http://www.youtube.com/watch?v=Ui86peQZ74s >>> >>> On Fri, Feb 26, 2010 at 3:26 PM, Noah Slater >> wrote: For the list's benefit, Randall sent me $2 via PayPal, suggesting I buy >> a candy bar. Thanks Randall, I may unofficially call this the Randall release. Kinda >> got a ring to it, that. If anyone else wants to sponsor the release, you know what to do. On 25 Feb 2010, at 20:25, Noah Slater wrote: > Send $1 to nsla...@tumbolia.org and I will cut it. > > On 25 Feb 2010, at 20:13, Randall Leeds wrote: > >> +1 >> Cut it! >> >> On Feb 25, 2010 12:08 PM, "Paul Davis" >> wrote: >> >>> So really all of the patches brought up in the last few hours are >> good >> candidates for 1.0. >> +1 > >> >> > > > -- > Filipe David Manana, > fdman...@gmail.com > PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B > > "Reasonable men adapt themselves to the world. > Unreasonable men adapt the world to themselves. > That's why all progress depends on unreasonable men."
Re: Windows 0.11 snapshot
On 26/02/2010 4:18 PM, Juhani Ränkimies wrote: The branch is an experiment, trying to find a solution for the compaction and db-delete problems on windows Ack - I should have guessed - sorry about that :) There is one patch specific to the windows build process; for the case when path to inno setup contains spaces. http://github.com/juranki/couchdb/commit/0d5ec88f08a0519fdf9521730361c6da0c3d4cb4 I've incorporated that one. Regarding the file versioning - I'm sorry that we haven't kept you in the loop a little better, but with Damien's guidance we think we have found a better strategy for this. The root of this strategy comes from a realization that if a file is opened on windows with FILE_SHARE_DELETE, the file can be deleted *or renamed* while it is open. One limitation is that a file of the same name can not be re-created while the old one still has handles open (the 'deleted but still open' file still appears in directory listings until the handle is closed, for example) Given this, what we can do is something like: * Arrange for erlang to be able to open the DB and view files with this flag. * Instead of deleting a file before replacing it, we first rename the file to a unique name (ie, based on a UUID) in a special directory. * As couch starts up, attempt to delete any old files in this special directory. In theory, no such files should exist - the OS should take care of actually removing any such files even if erlang crashes. The end result of this is that things can be made to work with a lot less friction than the 'file versioning' scheme. I've a patch to couchdb that works when used with a patch to erlang to open *all* files with that flag. The next step down this path seems to be to create an erlang 'port driver' for disk IO and use this instead of the builtin erlang file objects. We've identified a 'bfile' erlang port driver we may be able to fork and adapt to our needs - using our own port driver would also allow optimizations to the file IO for non-windows platforms (eg, indicate to the OS that certain files should not get OS-level caching, etc) Unfortunately, none of this has been discussed anywhere formal - much of it was on IRC - so I apologize if this is the first you heard of it. But whatever strategy we wind up with for this is almost certainly not going to land and be tested enough to make 0.11. Cheers, Mark
Re: Windows 0.11 snapshot
On 27/02/2010 12:31 AM, Noah Slater wrote: I'm cutting the release soon. Are you confident that I can do that with your artefacts? What day do you want to hand them over to me for the official vote? I thought we decided that the artefacts needed to be built by me from your signed copy of the tarball? I admit I have never tried to build from such a tarball (just an svn tree), so maybe I need to verify that first. Are there any such 0.11 tarballs I could use for a practice run? Cheers, Mark
bulk docs deletion issues
Hi, I just wanted to mention that the two issues I just created, COUCHDB-674 and COUCHDB-675, are not affecting real code. I was deliberately trying to get a not_found out of bulk docs to improve some tests; improved tests will have to wait for now ;-). - Matt
[jira] Created: (COUCHDB-675) Using _bulk_docs to delete a missing document gives success response
Using _bulk_docs to delete a missing document gives success response
Key: COUCHDB-675
URL: https://issues.apache.org/jira/browse/COUCHDB-675
Project: CouchDB
Issue Type: Bug
Components: Database Core
Reporter: Matt Goodall
Deleting a non-existant document using _bulk_docs should return not_found but
instead appears to work, returning a "1-xxx" deletion rev. The deletion even
appears in the changes.
$ curl http://localhost:5984/scratch -X PUT
{"ok":true}
$ curl http://localhost:5984/scratch/_bulk_docs -X POST -d '{"docs": [{"_id":
"foo", "_deleted": true}]}'
[{"id":"foo","rev":"1-de715bf126efc93d225d9e3f8e8eb81a"}]
$ curl http://localhost:5984/scratch/_changes
{"results":[
{"seq":1,"id":"foo","changes":[{"rev":"1-de715bf126efc93d225d9e3f8e8eb81a"}],"deleted":true}
],
"last_seq":1}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-674) Inconsistent error response, DELETE vs _bulk_docs deleted=true
Inconsistent error response, DELETE vs _bulk_docs deleted=true
--
Key: COUCHDB-674
URL: https://issues.apache.org/jira/browse/COUCHDB-674
Project: CouchDB
Issue Type: Bug
Components: Database Core
Reporter: Matt Goodall
The result from deleting an already deleted document using an HTTP DELETE and a
_bulk_docs {"deleted": true} is not consistent.
$ curl http://localhost:5984/scratch/foo -X PUT -d '{}'
{"ok":true,"id":"foo","rev":"1-967a00dff5e02add41819138abb3284d"}
$ curl http://localhost:5984/scratch/foo?rev=1-967a00dff5e02add41819138abb3284d
-X DELETE
{"ok":true,"id":"foo","rev":"2-eec205a9d413992850a6e32678485900"}
$ curl http://localhost:5984/scratch/foo?rev=1-967a00dff5e02add41819138abb3284d
-X DELETE
{"error":"not_found","reason":"deleted"}
$ curl http://localhost:5984/scratch/_bulk_docs -X POST -d '{"docs": [{"_id":
"foo", "_rev": "1-967a00dff5e02add41819138abb3284d", "_deleted": true}]}'
[{"id":"foo","error":"conflict","reason":"Document update conflict."}]
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: Releasing 0.11, Request for Comments
I would still like to see ticket 639 in 0.11. Adam told me, via IRC, he will review the patch by the end of this week. I vote +1 on it. cheers On Fri, Feb 26, 2010 at 3:44 PM, Noah Slater wrote: > Nope, you can send payment to any email. > > As long as the recipient can click on the link in the email, they can > deposit it into any account. > > On 26 Feb 2010, at 14:37, till wrote: > > > I always knew secretly open source worked like that. ;) > > > > Btw, I know it's not so subtle, but you need to include your paypal > email... :D > > > > http://www.youtube.com/watch?v=Ui86peQZ74s > > > > On Fri, Feb 26, 2010 at 3:26 PM, Noah Slater > wrote: > >> For the list's benefit, Randall sent me $2 via PayPal, suggesting I buy > a candy bar. > >> > >> Thanks Randall, I may unofficially call this the Randall release. Kinda > got a ring to it, that. > >> > >> If anyone else wants to sponsor the release, you know what to do. > >> > >> On 25 Feb 2010, at 20:25, Noah Slater wrote: > >> > >>> Send $1 to nsla...@tumbolia.org and I will cut it. > >>> > >>> On 25 Feb 2010, at 20:13, Randall Leeds wrote: > >>> > +1 > Cut it! > > On Feb 25, 2010 12:08 PM, "Paul Davis" > wrote: > > > So really all of the patches brought up in the last few hours are > good > candidates for 1.0. > +1 > >>> > >> > >> > > -- Filipe David Manana, fdman...@gmail.com PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Re: Releasing 0.11, Request for Comments
Nope, you can send payment to any email. As long as the recipient can click on the link in the email, they can deposit it into any account. On 26 Feb 2010, at 14:37, till wrote: > I always knew secretly open source worked like that. ;) > > Btw, I know it's not so subtle, but you need to include your paypal email... > :D > > http://www.youtube.com/watch?v=Ui86peQZ74s > > On Fri, Feb 26, 2010 at 3:26 PM, Noah Slater wrote: >> For the list's benefit, Randall sent me $2 via PayPal, suggesting I buy a >> candy bar. >> >> Thanks Randall, I may unofficially call this the Randall release. Kinda got >> a ring to it, that. >> >> If anyone else wants to sponsor the release, you know what to do. >> >> On 25 Feb 2010, at 20:25, Noah Slater wrote: >> >>> Send $1 to nsla...@tumbolia.org and I will cut it. >>> >>> On 25 Feb 2010, at 20:13, Randall Leeds wrote: >>> +1 Cut it! On Feb 25, 2010 12:08 PM, "Paul Davis" wrote: > So really all of the patches brought up in the last few hours are good candidates for 1.0. +1 >>> >> >>
Re: Releasing 0.11, Request for Comments
I always knew secretly open source worked like that. ;) Btw, I know it's not so subtle, but you need to include your paypal email... :D http://www.youtube.com/watch?v=Ui86peQZ74s On Fri, Feb 26, 2010 at 3:26 PM, Noah Slater wrote: > For the list's benefit, Randall sent me $2 via PayPal, suggesting I buy a > candy bar. > > Thanks Randall, I may unofficially call this the Randall release. Kinda got a > ring to it, that. > > If anyone else wants to sponsor the release, you know what to do. > > On 25 Feb 2010, at 20:25, Noah Slater wrote: > >> Send $1 to nsla...@tumbolia.org and I will cut it. >> >> On 25 Feb 2010, at 20:13, Randall Leeds wrote: >> >>> +1 >>> Cut it! >>> >>> On Feb 25, 2010 12:08 PM, "Paul Davis" wrote: >>> So really all of the patches brought up in the last few hours are good >>> candidates for 1.0. >>> +1 >> > >
Re: Releasing 0.11, Request for Comments
For the list's benefit, Randall sent me $2 via PayPal, suggesting I buy a candy bar. Thanks Randall, I may unofficially call this the Randall release. Kinda got a ring to it, that. If anyone else wants to sponsor the release, you know what to do. On 25 Feb 2010, at 20:25, Noah Slater wrote: > Send $1 to nsla...@tumbolia.org and I will cut it. > > On 25 Feb 2010, at 20:13, Randall Leeds wrote: > >> +1 >> Cut it! >> >> On Feb 25, 2010 12:08 PM, "Paul Davis" wrote: >> >>> So really all of the patches brought up in the last few hours are good >> candidates for 1.0. >> +1 >
Re: Windows 0.11 snapshot
I'm cutting the release soon. Are you confident that I can do that with your artefacts? What day do you want to hand them over to me for the official vote? On 26 Feb 2010, at 03:14, Mark Hammond wrote: > On 25/02/2010 10:38 PM, Noah Slater wrote: >> Might want to get this fixed for the release Mark. >> >> The release artefact should have an sha1 and md5 hash in the same directory. > > OK, done. I modified the build script to create the .sha file, and also to > arrange so both the sha and md5 files reference './setup-couchdb-xxx.exe > > New snapshots have been uploaded and are named couchdb-0.11.0b916031.* > > Cheers, > > Mark > >> >> Check the Verifying Releases section here: >> >> http://couchdb.apache.org/downloads.html >> >> Your files will need to pass this. >> >> On 25 Feb 2010, at 03:37, Mark Hammond wrote: >> >>> On 25/02/2010 1:51 PM, Nicholas Orr wrote: The md5 file is expecting the file to be in a dir called windows - might want to update the md5 :) >>> >>> The md5 is auto-generated by the Makefile in the 'etc' directory - I'll >>> make a fix for that as I find time (and patches welcome in the meantime ;) >>> >>> Cheers, >>> >>> Mark >> >
[jira] Commented: (COUCHDB-639) Make replication profit of attachment compression and improve push replication for large attachments
[ https://issues.apache.org/jira/browse/COUCHDB-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838855#action_12838855 ] Filipe Manana commented on COUCHDB-639: --- Just tested it with the latest trunk rev and found no problems: fdman...@core2duo:~/git/couchdb$ git log -1 commit 4cefde131f1992c70f66c527435a715290174423 Author: Mark Hammond Date: Fri Feb 26 01:32:21 2010 + generate .sha file for windows binary; ensure md5/sha use rel paths git-svn-id: https://svn.apache.org/repos/asf/couchdb/tr...@916528 13f79535-47bb-0310-9956-ffa450edef68 fdman...@core2duo:~/git/couchdb$ fdman...@core2duo:~/git/couchdb$ git apply --index --reject ../rep-att-comp-and-multipart-trunk-4.patch Checking patch src/couchdb/couch_db.erl... Checking patch src/couchdb/couch_doc.erl... Checking patch src/couchdb/couch_httpd_db.erl... Checking patch src/couchdb/couch_rep_att.erl... Checking patch src/couchdb/couch_rep_reader.erl... Checking patch src/couchdb/couch_rep_writer.erl... Checking patch test/etap/170-replication-attachment-comp.t... Checking patch test/etap/Makefile.am... Applied patch src/couchdb/couch_db.erl cleanly. Applied patch src/couchdb/couch_doc.erl cleanly. Applied patch src/couchdb/couch_httpd_db.erl cleanly. Applied patch src/couchdb/couch_rep_att.erl cleanly. Applied patch src/couchdb/couch_rep_reader.erl cleanly. Applied patch src/couchdb/couch_rep_writer.erl cleanly. Applied patch test/etap/170-replication-attachment-comp.t cleanly. Applied patch test/etap/Makefile.am cleanly. fdman...@core2duo:~/git/couchdb$ All tests are passing also. > Make replication profit of attachment compression and improve push > replication for large attachments > > > Key: COUCHDB-639 > URL: https://issues.apache.org/jira/browse/COUCHDB-639 > Project: CouchDB > Issue Type: Improvement > Components: Replication >Affects Versions: 0.11 > Environment: trunk >Reporter: Filipe Manana > Attachments: rep-att-comp-and-multipart-trunk-4.patch > > > At the moment, for compressed attachments, the replication uncompresses and > then compresses again the attachments. Therefore, a waste of CPU time. > The push replication is also not reliable for very large attachments (500mb + > for example). Currently it sends the attachments in-lined in the respective > JSON doc. Not only this requires too much ram memory, it also wastes too much > CPU time doing the base64 encoding of the attachment (and also a > decompression if the attachment is compressed). > The following patch (rep-att-comp-and-multipart-trunk*.patch) addresses both > issues. Docs containing attachments are now streamed to the target remote DB > using the multipart doc streaming feature provided by couch_doc.erl, and > compressed attachments are not uncompressed and re-compressed during the > replication > JavaScript tests included. > Previously doing a replication of a DB containing 2 docs with attachments of > 100mb and 500mb caused the Erlang VM to consume near 1.2GB of ram memory in > my system. With that patch applied, it uses about 130Mb of ram memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-549) include_docs=true doesn't honour conflicts=true
[ https://issues.apache.org/jira/browse/COUCHDB-549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Filipe Manana updated COUCHDB-549: -- Attachment: couchdb-549-trunk.patch Following patch fixes it. JS test included. cheers > include_docs=true doesn't honour conflicts=true > --- > > Key: COUCHDB-549 > URL: https://issues.apache.org/jira/browse/COUCHDB-549 > Project: CouchDB > Issue Type: Improvement > Components: HTTP Interface >Affects Versions: 0.11 >Reporter: Brian Candler >Priority: Minor > Attachments: couchdb-549-trunk.patch > > > When you read a view and use the option 'include_docs=true' to get the source > document in each result row, the option 'conflicts=true' is not honoured. You > do not see a _conflicts member in the document, even if it is in a > conflicting state. > This feature request could be expanded in a couple of directions: > 1. Make include_docs=true honour *all* options which a straightforward GET > would honour - e.g. revs, revs_info, open_revs. Maybe this would be > straightforward if they shared the same code path and options processing. > 2. It has been suggested that 'conflicts=true' could be the default anyway. > That is, whenever you retrieve a document, you get a _conflicts member if it > is in a conflicting state, without having to ask for it. This would be > unlikely to break things, but would make it less likely that conflicts would > go unnoticed, and it would simplify the API a little. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-639) Make replication profit of attachment compression and improve push replication for large attachments
[ https://issues.apache.org/jira/browse/COUCHDB-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838836#action_12838836 ] Filipe Manana commented on COUCHDB-639: --- Can't find which file? You should do: $ cd your_git_repo_path $ git apply rep-att-comp-and-multipart-trunk-4.patch > Make replication profit of attachment compression and improve push > replication for large attachments > > > Key: COUCHDB-639 > URL: https://issues.apache.org/jira/browse/COUCHDB-639 > Project: CouchDB > Issue Type: Improvement > Components: Replication >Affects Versions: 0.11 > Environment: trunk >Reporter: Filipe Manana > Attachments: rep-att-comp-and-multipart-trunk-4.patch > > > At the moment, for compressed attachments, the replication uncompresses and > then compresses again the attachments. Therefore, a waste of CPU time. > The push replication is also not reliable for very large attachments (500mb + > for example). Currently it sends the attachments in-lined in the respective > JSON doc. Not only this requires too much ram memory, it also wastes too much > CPU time doing the base64 encoding of the attachment (and also a > decompression if the attachment is compressed). > The following patch (rep-att-comp-and-multipart-trunk*.patch) addresses both > issues. Docs containing attachments are now streamed to the target remote DB > using the multipart doc streaming feature provided by couch_doc.erl, and > compressed attachments are not uncompressed and re-compressed during the > replication > JavaScript tests included. > Previously doing a replication of a DB containing 2 docs with attachments of > 100mb and 500mb caused the Erlang VM to consume near 1.2GB of ram memory in > my system. With that patch applied, it uses about 130Mb of ram memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-639) Make replication profit of attachment compression and improve push replication for large attachments
[ https://issues.apache.org/jira/browse/COUCHDB-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838833#action_12838833 ] sulantha sanjeewa commented on COUCHDB-639: --- Can any one please tell me how to install this patch.. ASAP please.. i have version 0.11.. it says it can't find the file > Make replication profit of attachment compression and improve push > replication for large attachments > > > Key: COUCHDB-639 > URL: https://issues.apache.org/jira/browse/COUCHDB-639 > Project: CouchDB > Issue Type: Improvement > Components: Replication >Affects Versions: 0.11 > Environment: trunk >Reporter: Filipe Manana > Attachments: rep-att-comp-and-multipart-trunk-4.patch > > > At the moment, for compressed attachments, the replication uncompresses and > then compresses again the attachments. Therefore, a waste of CPU time. > The push replication is also not reliable for very large attachments (500mb + > for example). Currently it sends the attachments in-lined in the respective > JSON doc. Not only this requires too much ram memory, it also wastes too much > CPU time doing the base64 encoding of the attachment (and also a > decompression if the attachment is compressed). > The following patch (rep-att-comp-and-multipart-trunk*.patch) addresses both > issues. Docs containing attachments are now streamed to the target remote DB > using the multipart doc streaming feature provided by couch_doc.erl, and > compressed attachments are not uncompressed and re-compressed during the > replication > JavaScript tests included. > Previously doing a replication of a DB containing 2 docs with attachments of > 100mb and 500mb caused the Erlang VM to consume near 1.2GB of ram memory in > my system. With that patch applied, it uses about 130Mb of ram memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-597) Replication tasks crash.
[
https://issues.apache.org/jira/browse/COUCHDB-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds updated COUCHDB-597:
--
Attachment: 0001-changes-replication-timeouts-and-att.-fixes-COUCHDB-.patch
Forgot to check the inclusion box.
> Replication tasks crash.
>
>
> Key: COUCHDB-597
> URL: https://issues.apache.org/jira/browse/COUCHDB-597
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.11
>Reporter: Robert Newson
> Attachments:
> 0001-changes-replication-timeouts-and-att.-fixes-COUCHDB-.patch,
> couchdb_597.patch
>
>
> If I kick off 10 replication tasks in quick succession, occasionally one or
> two of the replication tasks will die and not be resumed. It seems that the
> stat tracking is a little buggy, and under stress can eventually cause a
> permanent failure of the supervised replication task;
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [<0.80.0>] {error_report,<0.30.0>,
> {<0.80.0>,supervisor_report,
> [{supervisor,{local,couch_rep_sup}},
> {errorContext,shutdown_error},
> {reason,killed},
> {offender,
> [{pid,<0.6700.11>},
>{name,"fcbb13200a1618cf983b347f4d2c9835+create_target"},
>{mfa,
>{gen_server,start_link,
>[couch_rep,
> ["fcbb13200a1618cf983b347f4d2c9835",
> {[{<<"create_target">>,true},
>{<<"source">>,<<"http://node:5984/perf-p2";>>},
>{<<"target">>,<<"perf-p2">>}]},
> {user_ctx,null,[<<"_admin">>]}],
> []]}},
>{restart_type,temporary},
>{shutdown,1},
>{child_type,worker}]}]}}
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [emulator] Error in process
> <0.6705.11> with exit value:
> {badarg,[{ets,insert,[stats_hit_table,{{couchdb,open_os_files},-1}]},{couch_stats_collector,decrement,1}]}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-597) Replication tasks crash.
[
https://issues.apache.org/jira/browse/COUCHDB-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds updated COUCHDB-597:
--
Attachment: (was:
0001-changes-replication-timeouts-and-att.-fixes-COUCHDB-.patch)
> Replication tasks crash.
>
>
> Key: COUCHDB-597
> URL: https://issues.apache.org/jira/browse/COUCHDB-597
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.11
>Reporter: Robert Newson
> Attachments: couchdb_597.patch
>
>
> If I kick off 10 replication tasks in quick succession, occasionally one or
> two of the replication tasks will die and not be resumed. It seems that the
> stat tracking is a little buggy, and under stress can eventually cause a
> permanent failure of the supervised replication task;
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [<0.80.0>] {error_report,<0.30.0>,
> {<0.80.0>,supervisor_report,
> [{supervisor,{local,couch_rep_sup}},
> {errorContext,shutdown_error},
> {reason,killed},
> {offender,
> [{pid,<0.6700.11>},
>{name,"fcbb13200a1618cf983b347f4d2c9835+create_target"},
>{mfa,
>{gen_server,start_link,
>[couch_rep,
> ["fcbb13200a1618cf983b347f4d2c9835",
> {[{<<"create_target">>,true},
>{<<"source">>,<<"http://node:5984/perf-p2";>>},
>{<<"target">>,<<"perf-p2">>}]},
> {user_ctx,null,[<<"_admin">>]}],
> []]}},
>{restart_type,temporary},
>{shutdown,1},
>{child_type,worker}]}]}}
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [emulator] Error in process
> <0.6705.11> with exit value:
> {badarg,[{ets,insert,[stats_hit_table,{{couchdb,open_os_files},-1}]},{couch_stats_collector,decrement,1}]}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-597) Replication tasks crash.
[
https://issues.apache.org/jira/browse/COUCHDB-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds updated COUCHDB-597:
--
Attachment: 0001-changes-replication-timeouts-and-att.-fixes-COUCHDB-.patch
I went back and made this patch SUPER simple and straightforward.
Applies to the very most current trunk.
This should take no more than a minute to review; it's super simple now.
> Replication tasks crash.
>
>
> Key: COUCHDB-597
> URL: https://issues.apache.org/jira/browse/COUCHDB-597
> Project: CouchDB
> Issue Type: Bug
> Components: Database Core
>Affects Versions: 0.11
>Reporter: Robert Newson
> Attachments: couchdb_597.patch
>
>
> If I kick off 10 replication tasks in quick succession, occasionally one or
> two of the replication tasks will die and not be resumed. It seems that the
> stat tracking is a little buggy, and under stress can eventually cause a
> permanent failure of the supervised replication task;
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [<0.80.0>] {error_report,<0.30.0>,
> {<0.80.0>,supervisor_report,
> [{supervisor,{local,couch_rep_sup}},
> {errorContext,shutdown_error},
> {reason,killed},
> {offender,
> [{pid,<0.6700.11>},
>{name,"fcbb13200a1618cf983b347f4d2c9835+create_target"},
>{mfa,
>{gen_server,start_link,
>[couch_rep,
> ["fcbb13200a1618cf983b347f4d2c9835",
> {[{<<"create_target">>,true},
>{<<"source">>,<<"http://node:5984/perf-p2";>>},
>{<<"target">>,<<"perf-p2">>}]},
> {user_ctx,null,[<<"_admin">>]}],
> []]}},
>{restart_type,temporary},
>{shutdown,1},
>{child_type,worker}]}]}}
> [Fri, 11 Dec 2009 19:00:08 GMT] [error] [emulator] Error in process
> <0.6705.11> with exit value:
> {badarg,[{ets,insert,[stats_hit_table,{{couchdb,open_os_files},-1}]},{couch_stats_collector,decrement,1}]}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
