[jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13097067#comment-13097067 ] James Howe commented on COUCHDB-1252: - ETA on this patch Jason? A way to have views return _deleted documents - Key: COUCHDB-1252 URL: https://issues.apache.org/jira/browse/COUCHDB-1252 Project: CouchDB Issue Type: New Feature Components: JavaScript View Server Affects Versions: 1.1, 1.0.3 Reporter: James Howe Given that documents can be 'soft' deleted / deleted with auditing data by updating the document to include the _deleted property, it would be incredibly useful if there were a way to access these documents in a map function. Otherwise it is very difficult to find the auditing data - even more so if the original ids are unknown. I was thinking along the lines of a view query parameter 'include_deleted', but don't really mind how this is implemented, as long as it is there. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13097124#comment-13097124 ] Benoit Chesneau commented on COUCHDB-1270: -- I'm still reading the code. Just some another minor details, - couch_index_api.erl should probably better expose a behavior, so as anything that will be needed in indexers modules. So missing functions will be handled at compilation. - supervision. maybe couch_index_updater:start_link(self(), Mod), couch_index_compactor:start_link(self(), Mod), could be added as a child to a supervisor. So thet could eventually been restared using erlang supervision. Also just wanted to say I really like the current design. Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis Attachments: 0001-Minor-changes-for-new-indexing-engine.patch, 0002-Create-the-couch_index-application.patch, 0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis updated COUCHDB-1270: --- Attachment: (was: 0001-Minor-changes-for-new-indexing-engine.patch) Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis Attachments: 0002-Create-the-couch_index-application.patch, 0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis updated COUCHDB-1270: --- Attachment: (was: 0003-Create-the-couch_mrview-application.patch) Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis updated COUCHDB-1270: --- Attachment: (was: 0004-Remove-the-old-view-engine.patch) Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Joseph Davis updated COUCHDB-1270: --- Attachment: 0004-Remove-the-old-view-engine.patch 0003-Create-the-couch_mrview-application.patch 0002-Create-the-couch_index-application.patch 0001-Minor-changes-for-new-indexing-engine.patch @Filipe Good catch on both. For point one, I just removed the nil default. In my refactoring I undid almost all of these setting record fields to nil because its fairly silly to use a nonstandard value for undefined when the standard value is literally, undefined. One place I didn't change this yet was for the qserver variable because of how couch_query_servers:stop_doc_map is written and I was wanting to minimize the scope of these changes. As to 2, I added the file removal in the cancellation function. One thing this doesn't address is a generalized method for handling the new compaction lifetime model. We'll have to revisit this a bit to try and get a better abstraction for compaction. The timing tests I was running were against the views in [1] which you've used before for these types of tests. The test is just a simple time curl http://127.0.0.1:5984/indexer_test/_design/test/_view/view1?limit=0; or w/e the URL is for that design doc. Fairly consistently the new-views are a minute or two faster than trunk. So, nothing super improved but convincing enough to me that I haven't trashed performance. Also, I did spend a couple days profiling the view engine and comparing a couple dirty hacks. I'm convinced we can shorten these times even more with some more refactoring and playing with some other optimizations but they were leading me deep into the couch_query_servers and couch_os_process. I plan on backporting some refactoring for these parts next and that should set us up for some of the more advanced changes to the view engine. Trunk: 26m29.809s New-Views: 24m50.923s [1] http://fdmanana.couchone.com/indexer_test @Benoit I'd agree on turnign couch_index_api into a behavior but I decided to delay that until after we update the various applications and build system so that we have a proper ERL_LIBS directory. For behaviours to work they need to be importable and that's not the easiest given our current structure. Though it'll definitely be something to add in the future. For supervisors I think that could be a good idea, but I'm not the most familiar with them and their side effects to know if they'd be a good fit here. You mentioned on IRC that you'd take a swing at implementing that and I'd be quite interested to see what you come up witih. Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis Attachments: 0001-Minor-changes-for-new-indexing-engine.patch, 0002-Create-the-couch_index-application.patch, 0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
CouchDB performance
Blog: http://blog.mudynamics.com/2011/09/05/help-couchdb-break-the-c10k-barrier/ If you are interested in working on this, drop me a note and we can bump your concurrency on blitz.io to 10,000 for a week (first few to respond). I'm hoping that the configuration changes/patches dramatically improve CouchDB performance so we can all relax at scale. If you also have suggestions for me to try out (ERL_FLAGS, etc), happy to try it and report back. Thanks, K. --- http://blog.mudynamics.com http://blitz.io @pcapr
[jira] [Commented] (COUCHDB-1252) A way to have views return _deleted documents
[ https://issues.apache.org/jira/browse/COUCHDB-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13097657#comment-13097657 ] Jason Smith commented on COUCHDB-1252: -- Hi, James. There is an Iris Couch sprint starting Thursday September 8, running for one week. I expect to complete it during the sprint. A way to have views return _deleted documents - Key: COUCHDB-1252 URL: https://issues.apache.org/jira/browse/COUCHDB-1252 Project: CouchDB Issue Type: New Feature Components: JavaScript View Server Affects Versions: 1.1, 1.0.3 Reporter: James Howe Given that documents can be 'soft' deleted / deleted with auditing data by updating the document to include the _deleted property, it would be incredibly useful if there were a way to access these documents in a map function. Otherwise it is very difficult to find the auditing data - even more so if the original ids are unknown. I was thinking along the lines of a view query parameter 'include_deleted', but don't really mind how this is implemented, as long as it is there. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13097748#comment-13097748 ] Filipe Manana commented on COUCHDB-1270: Paul, All fine for me Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis Attachments: 0001-Minor-changes-for-new-indexing-engine.patch, 0002-Create-the-couch_index-application.patch, 0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
