Re: [jira] [Created] (COUCHDB-1252) A way to have views return _deleted documents
On Friday, August 19, 2011, Paul Davis wrote: > I don't think its an issue that the 404 is there. It *is* deleted after all. Though there's an argument to be made why we don't show the deleted body in the response. > > I would say that it should be a view option, but this will get a bit wonky for you. You'll need to see if *any* of the views have it included and then read the body from disk in the enum_changes_since loop. After that you'll have to filter it before passing it to the doc mapper. Also, I'm currently in the middle of rewriting the entire view engine so I wouldn't go too far if you end up needing to change lots of code. > > Also, you can't see them in a view because we've never had the request to write this patch. It wasn't *too* long ago that it was impossible to see _design docs in a view (and now that's even off by default). > > On Aug 18, 2011, at 11:07 PM, Jason Smith (JIRA) wrote: > >> >>[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087509#comment-13087509] >> >> Jason Smith commented on COUCHDB-1252: >> -- >> >> To Damien, I say: well, why can't I see deleted docs in a view? Why do I get a 404 when I query for them? Because they are deleted. Damien's code contradicts Damien's words, and the code makes better sense. To expose _deleted documents everywhere is a leaky abstraction. >> >> However, in the spirit of "design by committee is bad" and to learn a different part of CouchDB, I would like to write this patch. Paul, should it be a design option or a view option? >> >> For example: >> >> { "_id": "_design/example" >> , "options": { "these_are":"options global to the ddoc", "local_seq":true, "include_deleted":true} >> , "views": >> { "deleted_docs_A": >>{ "options": {"these_are":"local to this view", "include_design":true, "include_deleted":true} >>, "map":"function(D) { if(D._deleted) emit(D._id, 1) }" >>} >> , "deleted_docs_B": >>{ "options": {"not using include_deleted": true} >>, "map":"function(D) { if(D._deleted) emit(D._id, 1) /* Does it emit? */ } >>} >> } >> } >> >> With design options, deleted_docs_B will emit, but using view options, it won't. I prefer it to be per-view. In fact I did not even realize design options existed until I checked the source. >> >> Thoughts? Thanks. >> >>> A way to have views return _deleted documents >>> - >>> >>>Key: COUCHDB-1252 >>>URL: https://issues.apache.org/jira/browse/COUCHDB-1252 >>>Project: CouchDB >>> Issue Type: New Feature >>> Components: JavaScript View Server >>> Affects Versions: 1.1, 1.0.3 >>> Reporter: James Howe >>> >>> Given that documents can be 'soft' deleted / deleted with auditing data by updating the document to include the _deleted property, it would be incredibly useful if there were a way to access these documents in a map function. Otherwise it is very difficult to find the auditing data - even more so if the original ids are unknown. >>> I was thinking along the lines of a view query parameter 'include_deleted', but don't really mind how this is implemented, as long as it is there. >> >> -- >> This message is automatically generated by JIRA. >> For more information on JIRA, see: http://www.atlassian.com/software/jira >> >> > > maybe we should return a 410 instead of a 404 on _deleted but i'm n,ot sure it really worth it. 404 czn be a temporary response according the spec.
Re: [jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
I don't think its an issue that the 404 is there. It *is* deleted after all. Though there's an argument to be made why we don't show the deleted body in the response. I would say that it should be a view option, but this will get a bit wonky for you. You'll need to see if *any* of the views have it included and then read the body from disk in the enum_changes_since loop. After that you'll have to filter it before passing it to the doc mapper. Also, I'm currently in the middle of rewriting the entire view engine so I wouldn't go too far if you end up needing to change lots of code. Also, you can't see them in a view because we've never had the request to write this patch. It wasn't *too* long ago that it was impossible to see _design docs in a view (and now that's even off by default). On Aug 18, 2011, at 11:07 PM, Jason Smith (JIRA) wrote: > >[ > https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087509#comment-13087509 > ] > > Jason Smith commented on COUCHDB-1252: > -- > > To Damien, I say: well, why can't I see deleted docs in a view? Why do I get > a 404 when I query for them? Because they are deleted. Damien's code > contradicts Damien's words, and the code makes better sense. To expose > _deleted documents everywhere is a leaky abstraction. > > However, in the spirit of "design by committee is bad" and to learn a > different part of CouchDB, I would like to write this patch. Paul, should it > be a design option or a view option? > > For example: > > { "_id": "_design/example" > , "options": { "these_are":"options global to the ddoc", "local_seq":true, > "include_deleted":true} > , "views": > { "deleted_docs_A": >{ "options": {"these_are":"local to this view", "include_design":true, > "include_deleted":true} >, "map":"function(D) { if(D._deleted) emit(D._id, 1) }" >} > , "deleted_docs_B": >{ "options": {"not using include_deleted": true} >, "map":"function(D) { if(D._deleted) emit(D._id, 1) /* Does it emit? */ } >} > } > } > > With design options, deleted_docs_B will emit, but using view options, it > won't. I prefer it to be per-view. In fact I did not even realize design > options existed until I checked the source. > > Thoughts? Thanks. > >> A way to have views return _deleted documents >> - >> >>Key: COUCHDB-1252 >>URL: https://issues.apache.org/jira/browse/COUCHDB-1252 >>Project: CouchDB >> Issue Type: New Feature >> Components: JavaScript View Server >> Affects Versions: 1.1, 1.0.3 >> Reporter: James Howe >> >> Given that documents can be 'soft' deleted / deleted with auditing data by >> updating the document to include the _deleted property, it would be >> incredibly useful if there were a way to access these documents in a map >> function. Otherwise it is very difficult to find the auditing data - even >> more so if the original ids are unknown. >> I was thinking along the lines of a view query parameter 'include_deleted', >> but don't really mind how this is implemented, as long as it is there. > > -- > This message is automatically generated by JIRA. > For more information on JIRA, see: http://www.atlassian.com/software/jira > >
[jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087509#comment-13087509 ] Jason Smith commented on COUCHDB-1252: -- To Damien, I say: well, why can't I see deleted docs in a view? Why do I get a 404 when I query for them? Because they are deleted. Damien's code contradicts Damien's words, and the code makes better sense. To expose _deleted documents everywhere is a leaky abstraction. However, in the spirit of "design by committee is bad" and to learn a different part of CouchDB, I would like to write this patch. Paul, should it be a design option or a view option? For example: { "_id": "_design/example" , "options": { "these_are":"options global to the ddoc", "local_seq":true, "include_deleted":true} , "views": { "deleted_docs_A": { "options": {"these_are":"local to this view", "include_design":true, "include_deleted":true} , "map":"function(D) { if(D._deleted) emit(D._id, 1) }" } , "deleted_docs_B": { "options": {"not using include_deleted": true} , "map":"function(D) { if(D._deleted) emit(D._id, 1) /* Does it emit? */ } } } } With design options, deleted_docs_B will emit, but using view options, it won't. I prefer it to be per-view. In fact I did not even realize design options existed until I checked the source. Thoughts? Thanks. > A way to have views return _deleted documents > - > > Key: COUCHDB-1252 > URL: https://issues.apache.org/jira/browse/COUCHDB-1252 > Project: CouchDB > Issue Type: New Feature > Components: JavaScript View Server >Affects Versions: 1.1, 1.0.3 >Reporter: James Howe > > Given that documents can be 'soft' deleted / deleted with auditing data by > updating the document to include the _deleted property, it would be > incredibly useful if there were a way to access these documents in a map > function. Otherwise it is very difficult to find the auditing data - even > more so if the original ids are unknown. > I was thinking along the lines of a view query parameter 'include_deleted', > but don't really mind how this is implemented, as long as it is there. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087489#comment-13087489 ] Paul Joseph Davis commented on COUCHDB-1252: @Jason, A couple weeks ago I would have agreed completely with you. So much in fact that when someone reported a bug that _bulk_docs was storing the body of docs submitted with _deleted: true set, it was immediately assumed to be a bug and was promptly fixed. Until Damien said "That was on purpose so people can say why something was deleted." In reality, the only "special" sauce for deleted docs is that they're not considered conflicts. Other than that, you can still create a doc based on the revision of a deleted doc. In fact you can rescue an edit history from any known revision. But I digress. Either way, "auditing information about deleted docs" is apparently precisely what deleted doc bodies were for, so I don't see a reason to not have an option to see them. > A way to have views return _deleted documents > - > > Key: COUCHDB-1252 > URL: https://issues.apache.org/jira/browse/COUCHDB-1252 > Project: CouchDB > Issue Type: New Feature > Components: JavaScript View Server >Affects Versions: 1.1, 1.0.3 >Reporter: James Howe > > Given that documents can be 'soft' deleted / deleted with auditing data by > updating the document to include the _deleted property, it would be > incredibly useful if there were a way to access these documents in a map > function. Otherwise it is very difficult to find the auditing data - even > more so if the original ids are unknown. > I was thinking along the lines of a view query parameter 'include_deleted', > but don't really mind how this is implemented, as long as it is there. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087485#comment-13087485 ] Jason Smith commented on COUCHDB-1252: -- I propose this issue be closed as NOFIX (or whatever it's called). Allowing include_deleted would confuse the concept of "deleted" in the way that calling them "revisions" implies that CouchDB revisions are great for a wiki. James, your requirements are brilliant. That sounds like a great app. But IMO if you need data to remain indefinitely (for audit), then you should not be deleting documents and then exploiting the fact that CouchDB keeps deleted documents for revision tracking. In your application, "deleted" means "invisible to users but the auditors still want to know about it" so that is an application-level detail. That is exactly what views are for. For an application, _deleted means it's dead. Advanced tools like the replicator must follow the doc into the afterlife but that's not an application's concern. > A way to have views return _deleted documents > - > > Key: COUCHDB-1252 > URL: https://issues.apache.org/jira/browse/COUCHDB-1252 > Project: CouchDB > Issue Type: New Feature > Components: JavaScript View Server >Affects Versions: 1.1, 1.0.3 >Reporter: James Howe > > Given that documents can be 'soft' deleted / deleted with auditing data by > updating the document to include the _deleted property, it would be > incredibly useful if there were a way to access these documents in a map > function. Otherwise it is very difficult to find the auditing data - even > more so if the original ids are unknown. > I was thinking along the lines of a view query parameter 'include_deleted', > but don't really mind how this is implemented, as long as it is there. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-822) body_too_large error for external processes, when body is 3MB
[ https://issues.apache.org/jira/browse/COUCHDB-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087144#comment-13087144 ] Filipe Manana commented on COUCHDB-822: --- I think the right approach here is something like this: https://github.com/apache/couchdb/commit/cc2379ebbd45d842204642ac8a6a5a16669ffa4e The case where the request doesn't have neither a content-length neither is chunked encoded, should be very rare (not http compliant). Therefore it doesn't make sense to have a new setting. > body_too_large error for external processes, when body is 3MB > - > > Key: COUCHDB-822 > URL: https://issues.apache.org/jira/browse/COUCHDB-822 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Affects Versions: 0.11 > Environment: CouchDB 0.11 on Ubuntu 10.04, >Reporter: Erik Edin >Assignee: Robert Newson >Priority: Minor > Fix For: 1.2 > > > I have a photo which is around 3 MB that I'm trying to PUT to an external > process on the CouchDB database. The external process is called "_upload". > I get an uncaught error "{exit,{body_too_large,content_length}}" in the logs > when trying this. Smaller photos (around 60 kB) seem to work just fine. > This just happens with external processes. I can upload the photo as an > attachment directly to a document, with no problems. > The error is similar to an earlier bug in the mochiweb library that was fixed > around Feb 2009, where mochiweb never used the "max_document_size" setting > that was provided when calling mochiweb_request:recv_body. > I believe, supported by the stack trace below, that the cause for this bug is > that in couch_httpd_external:json_req_obj the function > mochiweb_request:recv_body/0 is called, which uses the mochiweb default value > on MaxBody, which is 1 MB. > I think that couch_httpd_external:json_req_obj should call > mochiweb_request:recv_body/1 instead, with the "max_document_size" setting as > the argument. > Here are the error logs from one of my attempts: > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] 'PUT' > /pillowfight/_upload/6b1908c352129ddda396fa69ac003d11 {1,1} > Headers: [{'Accept',"*/*"}, > {'Content-Length',"3093976"}, > {'Content-Type',"image/jpg"}, > {"Expect","100-continue"}, > {'Host',"localhost:5984"}, > {'User-Agent',"curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 > OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15"}] > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] OAuth Params: [] > [Thu, 08 Jul 2010 18:49:53 GMT] [error] [<0.3738.0>] Uncaught error in HTTP > request: {exit,{body_too_large,content_length}} > [Thu, 08 Jul 2010 18:49:53 GMT] [info] [<0.3738.0>] Stacktrace: > [{mochiweb_request,stream_body,5}, > {mochiweb_request,recv_body,2}, > {couch_httpd_external,json_req_obj,3}, > {couch_httpd_external,process_external_req,3}, > {couch_httpd_db,do_db_req,2}, > {couch_httpd,handle_request_int,5}, > {mochiweb_http,headers,5}, > {proc_lib,init_p_do_apply,3}] -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (COUCHDB-822) body_too_large error for external processes, when body is 3MB
[ https://issues.apache.org/jira/browse/COUCHDB-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Chesneau reopened COUCHDB-822: - The max external body size should have its proper settings rather than using the max doc size. Different usages. > body_too_large error for external processes, when body is 3MB > - > > Key: COUCHDB-822 > URL: https://issues.apache.org/jira/browse/COUCHDB-822 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Affects Versions: 0.11 > Environment: CouchDB 0.11 on Ubuntu 10.04, >Reporter: Erik Edin >Assignee: Robert Newson >Priority: Minor > Fix For: 1.2 > > > I have a photo which is around 3 MB that I'm trying to PUT to an external > process on the CouchDB database. The external process is called "_upload". > I get an uncaught error "{exit,{body_too_large,content_length}}" in the logs > when trying this. Smaller photos (around 60 kB) seem to work just fine. > This just happens with external processes. I can upload the photo as an > attachment directly to a document, with no problems. > The error is similar to an earlier bug in the mochiweb library that was fixed > around Feb 2009, where mochiweb never used the "max_document_size" setting > that was provided when calling mochiweb_request:recv_body. > I believe, supported by the stack trace below, that the cause for this bug is > that in couch_httpd_external:json_req_obj the function > mochiweb_request:recv_body/0 is called, which uses the mochiweb default value > on MaxBody, which is 1 MB. > I think that couch_httpd_external:json_req_obj should call > mochiweb_request:recv_body/1 instead, with the "max_document_size" setting as > the argument. > Here are the error logs from one of my attempts: > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] 'PUT' > /pillowfight/_upload/6b1908c352129ddda396fa69ac003d11 {1,1} > Headers: [{'Accept',"*/*"}, > {'Content-Length',"3093976"}, > {'Content-Type',"image/jpg"}, > {"Expect","100-continue"}, > {'Host',"localhost:5984"}, > {'User-Agent',"curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 > OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15"}] > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] OAuth Params: [] > [Thu, 08 Jul 2010 18:49:53 GMT] [error] [<0.3738.0>] Uncaught error in HTTP > request: {exit,{body_too_large,content_length}} > [Thu, 08 Jul 2010 18:49:53 GMT] [info] [<0.3738.0>] Stacktrace: > [{mochiweb_request,stream_body,5}, > {mochiweb_request,recv_body,2}, > {couch_httpd_external,json_req_obj,3}, > {couch_httpd_external,process_external_req,3}, > {couch_httpd_db,do_db_req,2}, > {couch_httpd,handle_request_int,5}, > {mochiweb_http,headers,5}, > {proc_lib,init_p_do_apply,3}] -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-822) body_too_large error for external processes, when body is 3MB
[ https://issues.apache.org/jira/browse/COUCHDB-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Newson resolved COUCHDB-822. --- Resolution: Fixed Fix Version/s: 1.2 Assignee: Robert Newson > body_too_large error for external processes, when body is 3MB > - > > Key: COUCHDB-822 > URL: https://issues.apache.org/jira/browse/COUCHDB-822 > Project: CouchDB > Issue Type: Bug > Components: HTTP Interface >Affects Versions: 0.11 > Environment: CouchDB 0.11 on Ubuntu 10.04, >Reporter: Erik Edin >Assignee: Robert Newson >Priority: Minor > Fix For: 1.2 > > > I have a photo which is around 3 MB that I'm trying to PUT to an external > process on the CouchDB database. The external process is called "_upload". > I get an uncaught error "{exit,{body_too_large,content_length}}" in the logs > when trying this. Smaller photos (around 60 kB) seem to work just fine. > This just happens with external processes. I can upload the photo as an > attachment directly to a document, with no problems. > The error is similar to an earlier bug in the mochiweb library that was fixed > around Feb 2009, where mochiweb never used the "max_document_size" setting > that was provided when calling mochiweb_request:recv_body. > I believe, supported by the stack trace below, that the cause for this bug is > that in couch_httpd_external:json_req_obj the function > mochiweb_request:recv_body/0 is called, which uses the mochiweb default value > on MaxBody, which is 1 MB. > I think that couch_httpd_external:json_req_obj should call > mochiweb_request:recv_body/1 instead, with the "max_document_size" setting as > the argument. > Here are the error logs from one of my attempts: > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] 'PUT' > /pillowfight/_upload/6b1908c352129ddda396fa69ac003d11 {1,1} > Headers: [{'Accept',"*/*"}, > {'Content-Length',"3093976"}, > {'Content-Type',"image/jpg"}, > {"Expect","100-continue"}, > {'Host',"localhost:5984"}, > {'User-Agent',"curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 > OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15"}] > [Thu, 08 Jul 2010 18:49:53 GMT] [debug] [<0.3738.0>] OAuth Params: [] > [Thu, 08 Jul 2010 18:49:53 GMT] [error] [<0.3738.0>] Uncaught error in HTTP > request: {exit,{body_too_large,content_length}} > [Thu, 08 Jul 2010 18:49:53 GMT] [info] [<0.3738.0>] Stacktrace: > [{mochiweb_request,stream_body,5}, > {mochiweb_request,recv_body,2}, > {couch_httpd_external,json_req_obj,3}, > {couch_httpd_external,process_external_req,3}, > {couch_httpd_db,do_db_req,2}, > {couch_httpd,handle_request_int,5}, > {mochiweb_http,headers,5}, > {proc_lib,init_p_do_apply,3}] -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1253) Resetting config chain not working properly
[ https://issues.apache.org/jira/browse/COUCHDB-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087056#comment-13087056 ] Noah Slater commented on COUCHDB-1253: -- Please change the top line of the couchdb script from: #! /bin/sh -e To: #! /bin/sh -ex Then run the command again, piping the output to a file: couchdb -n -a /tmp/tmpuPqqSF.paisley.test/test.ini &> output.txt Then attach the file to this ticket. Thanks. > Resetting config chain not working properly > --- > > Key: COUCHDB-1253 > URL: https://issues.apache.org/jira/browse/COUCHDB-1253 > Project: CouchDB > Issue Type: Bug > Components: Database Core >Affects Versions: 1.1 > Environment: Linux 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 > 21:09:46 UTC 2011 i686 GNU/Linux > Ubuntu 10.04.3 LTS >Reporter: Daniel Gonzalez >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Resetting the config chain seems to be working: > couchdb -n -a /tmp/tmpuPqqSF.paisley.test/test.ini -c > /tmp/tmpuPqqSF.paisley.test/test.ini > But it does actually not work: > couchdb -n -a /tmp/tmpuPqqSF.paisley.test/test.ini > ... > =INFO REPORT 18-Aug-2011::15:21:32 === > application: couch > exited: {bad_return,{{couch_app,start, > [normal, > ["/usr/local/etc/couchdb/default.ini", > "/usr/local/etc/couchdb/local.ini"]]}, > {'EXIT',{{badmatch,{error,shutdown}}, > [{couch_server_sup,start_server,1}, >{application_master,start_it_old,4}] > type: temporary > ... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1253) Resetting config chain not working properly
Resetting config chain not working properly --- Key: COUCHDB-1253 URL: https://issues.apache.org/jira/browse/COUCHDB-1253 Project: CouchDB Issue Type: Bug Components: Database Core Affects Versions: 1.1 Environment: Linux 2.6.32-33-generic #70-Ubuntu SMP Thu Jul 7 21:09:46 UTC 2011 i686 GNU/Linux Ubuntu 10.04.3 LTS Reporter: Daniel Gonzalez Priority: Minor Resetting the config chain seems to be working: couchdb -n -a /tmp/tmpuPqqSF.paisley.test/test.ini -c /tmp/tmpuPqqSF.paisley.test/test.ini But it does actually not work: couchdb -n -a /tmp/tmpuPqqSF.paisley.test/test.ini ... =INFO REPORT 18-Aug-2011::15:21:32 === application: couch exited: {bad_return,{{couch_app,start, [normal, ["/usr/local/etc/couchdb/default.ini", "/usr/local/etc/couchdb/local.ini"]]}, {'EXIT',{{badmatch,{error,shutdown}}, [{couch_server_sup,start_server,1}, {application_master,start_it_old,4}] type: temporary ... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira