Re: replicator options
On Sun, Jan 24, 2010 at 2:11 PM, Chris Anderson wrote: > On Sun, Jan 24, 2010 at 2:04 PM, Glenn Rempe wrote: > > On Sun, Jan 24, 2010 at 12:09 AM, Chris Anderson > wrote: > > > >> Devs, > >> > >> I've been thinking there are a few simple options that would magnify > >> the power of the replicator a lot. > >> > >> ... > >> The fun one is chained map reduce. It occurred to me the other night > >> that simplest way to present a chainable map reduce abstraction to > >> users is through the replicator. The action "copy these view rows to a > >> new db" is a natural fit for the replicator. I imagine this would be > >> super useful to people doing big messy data munging, and it wouldn't > >> be too hard for the replicator to handle. > >> > >> > > I like this idea as well, as chainable map/reduce has been something I > think > > a lot of people would like to use. The thing I am concerned about, and > > which is related to another ongoing thread, is the size of views on disk > and > > the slowness of generating them. I fear that we would end up ballooning > > views on disk to a size that is unmanageable if we chained them. I have > an > > app in production with 50m rows, whose DB has grown to >100GB, and the > views > > take up approx 800GB (!). I don't think I could afford the disk space to > > even consider using this especially when you consider that in order to > > compact a DB or view you need roughly 2x the disk space of the files on > > disk. > > > > I also worry about the time to generate chained views, when the time > needed > > for generating views currently is already a major weak point of CouchDB > > (Generating my views took more than a week). > > > > In practice, I think only those with relatively small DB's would be able > to > > take advantage of this feature. > > > > For large data, you'll want a cluster. The same holds true for other > Map Reduce frameworks like Hadoop or Google's stuff. > > That would not resolve the issue I mentioned where views can be a multiple in size of the original data DB. I have about 9 views in a design doc, and my resultant view files on disk are about 9x the size of the original DB data. How would sharding this across multiple DBs in a cluster resolve this? You would still end up with views that are some multiple in size of their original sharded DB. Compounded by how many replicas you have of that view data for chained M/R. > I'd be interested if anyone with partitioned CouchDB query experience > (Lounger or otherwise) can comment on view generation time when > parallelized across multiple machines. > > I would also be interested in seeing any architectures that make use of this to parallelize view generation. I'm not sure your example of Hadoop or Google M/R are really valid because they provide file system abstractions (e.g. Hadoop FS) for automatically streaming a single copy of the data to where it is needed to be Mapped/Reduced and CouchDB has nothing similar. http://hadoop.apache.org/common/docs/current/hdfs_design.html Don't get me wrong, I would love to see these things happen, I just wonder if there are other issues that need to be resolved first before this is practical for anything but a small dataset.
Re: replicator options
On Sun, Jan 24, 2010 at 12:09 AM, Chris Anderson wrote: > Devs, > > I've been thinking there are a few simple options that would magnify > the power of the replicator a lot. > > ... > The fun one is chained map reduce. It occurred to me the other night > that simplest way to present a chainable map reduce abstraction to > users is through the replicator. The action "copy these view rows to a > new db" is a natural fit for the replicator. I imagine this would be > super useful to people doing big messy data munging, and it wouldn't > be too hard for the replicator to handle. > > I like this idea as well, as chainable map/reduce has been something I think a lot of people would like to use. The thing I am concerned about, and which is related to another ongoing thread, is the size of views on disk and the slowness of generating them. I fear that we would end up ballooning views on disk to a size that is unmanageable if we chained them. I have an app in production with 50m rows, whose DB has grown to >100GB, and the views take up approx 800GB (!). I don't think I could afford the disk space to even consider using this especially when you consider that in order to compact a DB or view you need roughly 2x the disk space of the files on disk. I also worry about the time to generate chained views, when the time needed for generating views currently is already a major weak point of CouchDB (Generating my views took more than a week). In practice, I think only those with relatively small DB's would be able to take advantage of this feature.
Re: authentication cleanup
On Sun, Dec 27, 2009 at 11:46 AM, Benoit Chesneau wrote: > On Sun, Dec 27, 2009 at 8:05 PM, Chris Anderson wrote: > > > > Not really since salt is available and hash is only sha1. I think we > could make it harder but I agree with a strong encryption we don't > need to hide them. > > > > I'd be happy if someone can work out a stronger cryptographic > > solution. I don't think that there's much we can do to make brute > > force password guessing harder (aside from hiding the user's db, which > > we should do), but I'd be happy to be shown otherwise. > > Maybe we could start by using sha256. or more. I will have a look on > what could be done about it. > > I am just jumping in late here, so forgive me if this has been discussed. What about using bcrypt? Which is *designed* to be a slow hashing algorithm so that you dramatically reduce the ability to conduct brute force attacks. sha1, sha256, etc are designed to be fast, which if you gain access to the raw hashes is a bad thing. The idea is to make brute force attacks as computationally expensive as possible, while not impeding normal usage. for general context see : http://bit.ly/7nR09X http://bit.ly/91a05S And it seems like Adam K. has brought this up before as an option: http://bit.ly/7nxlA1
[jira] Commented: (COUCHDB-592) Build and configure without the js library
[ https://issues.apache.org/jira/browse/COUCHDB-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786150#action_12786150 ] Glenn Rempe commented on COUCHDB-592: - Noah, I believe you can use CouchDB without http access if you use the Hovercraft Erlang library. http://github.com/jchris/hovercraft http://github.com/grempe/hovercraft (Includes fixes for trunk couchdb pending merge to jchris master) If you mixed that with Erlang views wouldn't that be a reason to want to eliminate any external build time dependencies other than Erlang? CouchDB should optimally be able to operate in pure erlang mode. No? This is something the couchdb team should encourage imho. > Build and configure without the js library > -- > > Key: COUCHDB-592 > URL: https://issues.apache.org/jira/browse/COUCHDB-592 > Project: CouchDB > Issue Type: New Feature > Components: Build System > Environment: Any? >Reporter: Ari Lerner > > I am trying to use couchdb without the requirements of the js library. I will > not be accessing couch (nor do I want access) through an http interface, but > an erlang client library. > Thanks -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-583) adding ?compression=(gzip|deflate) optional parameter to the attachment download API
[ https://issues.apache.org/jira/browse/COUCHDB-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783495#action_12783495 ] Glenn Rempe commented on COUCHDB-583: - As a side note (this should probably be a discussion of its own though). Has any thought been given to bolting WebMachine onto CouchDB as the primary HTTP interface? I'm not sure how difficult it would be to retrofit, or if there are use cases that would preclude this, but doing so would seem to eliminate a lot of uncertainty (and possible errors) in making sure that the complexities of HTTP are properly handled? I comment on it here since webmachine could in theory 'do the right thing' regarding handling content encoding, HTTP response codes, etc. http://bitbucket.org/justin/webmachine/wiki/Home Just a thought. I'm curious if this has been discussed before. > adding ?compression=(gzip|deflate) optional parameter to the attachment > download API > > > Key: COUCHDB-583 > URL: https://issues.apache.org/jira/browse/COUCHDB-583 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface > Environment: CouchDB trunk revision 885240 >Reporter: Filipe Manana > Attachments: jira-couchdb-583-1st-try-trunk.patch, > jira-couchdb-583-2nd-try-trunk.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > The following new feature is added in the patch following this ticket > creation. > A new optional http query parameter "compression" is added to the attachments > API. > This parameter can have one of the values: "gzip" or "deflate". > When asking for an attachment (GET http request), if the query parameter > "compression" is found, CouchDB will send the attachment compressed to the > client (and sets the header Content-Encoding with gzip or deflate). > Further, it adds a new config option "treshold_for_chunking_comp_responses" > (httpd section) that specifies an attachment length threshold. If an > attachment has a length >= than this threshold, the http response will be > chunked (besides compressed). > Note that using non chunked compressed body responses requires storing all > the compressed blocks in memory and then sending each one to the client. This > is a necessary "evil", as we only know the length of the compressed body > after compressing all the body, and we need to set the "Content-Length" > header for non chunked responses. By sending chunked responses, we can send > each compressed block immediately, without accumulating all of them in memory. > Examples: > $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=gzip > $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=deflate > $ curl http://localhost:5984/testdb/testdoc1/readme.txt # attachment will > not be compressed > $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=rar # > will give a 500 error code > Etap test case included. > Feedback would be very welcome. > cheers -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Hovercraft : Updates and Help Needed
PS - What do you think about getting Hovercraft built into the CouchDB source? Seems like it would be a natural addition and allow everyone to get a native Erlang driver for free. Now that it has tests that run green it would hopefully get some more developer love and stay in sync with future releases of CouchDB (and failures due to changes in CouchDB would jump out instead of discovery later as hovercraft naturally lags behind trunk). Cheers, Glenn
Re: Hovercraft : Updates and Help Needed
Hi Chris, Great news! I sent you a pull request, but for the benefit of others, I made some more commits today, learned some more Erlang along the way, and got the previous test suite for Hovercraft running 100% green! I must say, I was very pleased. I did note though that in the middle of my hackathon you also dropped in a few commits yourself. :-) Please feel free to merge my code in and choose the bits that help. I didn't yet look in detail at your changes, but I'll look tomorrow and compare to see how we differed in attacking the problem. Comments on my changes are of course welcome. My forked branch with my latest updates is here: http://github.com/grempe/hovercraft/tree/fixes Cheers, Glenn On Sun, Nov 8, 2009 at 1:47 PM, Chris Anderson wrote: > On Sat, Nov 7, 2009 at 2:44 PM, Glenn Rempe wrote: >> Hi, >> >> Over the last day I have forked and made a number of changes to the >> Hovercraft Erlang CouchDB lib. As I am pretty new to Erlang, the >> changes are mostly in the realm of cleanup and improved documentation. >> I did however fix a couple of issues with getting hovercraft working >> with the current trunk CouchDB. This has been a great way for me to >> contribute (I hope) as well as to learn more about how hovercraft >> works. >> >> I have identified three specific issues that are preventing the >> current test suite from running cleanly and documented those problems >> in my fork on GitHub. I am hoping that someone here on the dev list >> with much more knowledge of couchdb/erlang and recent refactoring in >> trunk can take a look at these issues and help me patch them. Once >> done I would love to get all the changes pulled into the main repo >> that jchris owns. Note that my fork is actually off of 'sendtopms's >> fork, which fixed some issues in jchris' code with current trunk. >> >> The changes are in the 'fixes' branch of my fork which is here: >> >> http://github.com/grempe/hovercraft/tree/fixes >> >> If you take a look at the TODO section of the README.md file you will >> see the specific breaking issues and the errors that are being thrown. >> >> The README with the TODO section with issues that need attentions is here: >> >> http://github.com/grempe/hovercraft/blob/fixes/README.md >> >> FYI, here is a summary of the changes I have made so far: >> >> - couch_util:new_uuid() is now couch_uuids:random() in trunk, fixed >> this in hovercraft >> - Cleaned up and enhanced the README docs. >> - Fixed some documentation issues in the comments of the hovercraft.erl file >> - Broke out the test suite into its own (hovercraft_test) module, and >> renamed a couple of the public test functions to match this >> namespacing. >> - commented out and marked with FIXME two failing tests in >> hovercraft_test:all/0 >> - Marked with FIXME a call in hovercraft:query which calls a >> non-exported couch_view:fold_reduce/7 >> - fixed the include statement so that hovercraft can be compiled as >> shown in the docs >> >> Thanks for the help. Hopefully with some help we can get hovercraft >> cleanly working with trunk and passing all its tests. >> > > Thanks for bringing this to my attention Glenn. > > I've pulled your fixes branch, and helped it along a little bit (map > views work, reduce works, but group_level is still broken). I also > haven't started on the attachment stuff. > > Anyone should feel free to pitch in. The latest shiny fixed version is at: > > http://github.com/jchris/hovercraft/tree/fixes > > Chris > > > -- > Chris Anderson > http://jchrisa.net > http://couch.io > -- Glenn Rempe email : gl...@rempe.us voice : (415) 894-5366 or (415)-89G-LENN twitter: @grempe contact info: http://www.rempe.us/contact.html pgp: http://www.rempe.us/gnupg.txt
Hovercraft : Updates and Help Needed
Hi, Over the last day I have forked and made a number of changes to the Hovercraft Erlang CouchDB lib. As I am pretty new to Erlang, the changes are mostly in the realm of cleanup and improved documentation. I did however fix a couple of issues with getting hovercraft working with the current trunk CouchDB. This has been a great way for me to contribute (I hope) as well as to learn more about how hovercraft works. I have identified three specific issues that are preventing the current test suite from running cleanly and documented those problems in my fork on GitHub. I am hoping that someone here on the dev list with much more knowledge of couchdb/erlang and recent refactoring in trunk can take a look at these issues and help me patch them. Once done I would love to get all the changes pulled into the main repo that jchris owns. Note that my fork is actually off of 'sendtopms's fork, which fixed some issues in jchris' code with current trunk. The changes are in the 'fixes' branch of my fork which is here: http://github.com/grempe/hovercraft/tree/fixes If you take a look at the TODO section of the README.md file you will see the specific breaking issues and the errors that are being thrown. The README with the TODO section with issues that need attentions is here: http://github.com/grempe/hovercraft/blob/fixes/README.md FYI, here is a summary of the changes I have made so far: - couch_util:new_uuid() is now couch_uuids:random() in trunk, fixed this in hovercraft - Cleaned up and enhanced the README docs. - Fixed some documentation issues in the comments of the hovercraft.erl file - Broke out the test suite into its own (hovercraft_test) module, and renamed a couple of the public test functions to match this namespacing. - commented out and marked with FIXME two failing tests in hovercraft_test:all/0 - Marked with FIXME a call in hovercraft:query which calls a non-exported couch_view:fold_reduce/7 - fixed the include statement so that hovercraft can be compiled as shown in the docs Thanks for the help. Hopefully with some help we can get hovercraft cleanly working with trunk and passing all its tests. Cheers, Glenn
[jira] Created: (COUCHDB-563) Typo in couch_view_updater.erl
Typo in couch_view_updater.erl
--
Key: COUCHDB-563
URL: https://issues.apache.org/jira/browse/COUCHDB-563
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 0.11
Reporter: Glenn Rempe
Priority: Minor
There is a typo on three lines in couch_view_updater.erl where the Var name
'IntitalBuild' should be 'InitialBuild'
diff --git a/src/couchdb/couch_view_updater.erl
b/src/couchdb/couch_view_updater.erl
index 6f04d93..475f59a 100644
--- a/src/couchdb/couch_view_updater.erl
+++ b/src/couchdb/couch_view_updater.erl
@@ -137,7 +137,7 @@ do_maps(Group, MapQueue, WriteQueue, ViewEmptyKVs) ->
do_maps(Group1, MapQueue, WriteQueue, ViewEmptyKVs)
end.
-do_writes(Parent, Owner, Group, WriteQueue, IntitalBuild) ->
+do_writes(Parent, Owner, Group, WriteQueue, InitialBuild) ->
case couch_work_queue:dequeue(WriteQueue) of
closed ->
Parent ! {new_group, Group};
@@ -155,12 +155,12 @@ do_writes(Parent, Owner, Group, WriteQueue,
IntitalBuild) ->
AccViewKVs2, DocIdViewIdKeys ++ AccDocIdViewIdKeys}
end, nil, Queue),
Group2 = write_changes(Group, ViewKeyValues, DocIdViewIdKeys, NewSeq,
-IntitalBuild),
+InitialBuild),
case Owner of
nil -> ok;
_ -> ok = gen_server:cast(Owner, {partial_update, Parent, Group2})
end,
-do_writes(Parent, Owner, Group2, WriteQueue, IntitalBuild)
+do_writes(Parent, Owner, Group2, WriteQueue, InitialBuild)
end.
view_insert_query_results([], [], ViewKVs, DocIdViewIdKeysAcc) ->
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-550) Provide Futon UI option that would use 'stale=ok' when browsing DB views in Futon, useful for write-heavy DB's
Provide Futon UI option that would use 'stale=ok' when browsing DB views in
Futon, useful for write-heavy DB's
--
Key: COUCHDB-550
URL: https://issues.apache.org/jira/browse/COUCHDB-550
Project: CouchDB
Issue Type: Improvement
Components: Administration Console
Environment: All
Reporter: Glenn Rempe
If a CouchDB has continuous writes, as in the case of a log capturing system,
it becomes very difficult to use Futon to browse the views in a design
document. If writes are continuous in the background, and the admin chooses a
view to browse in Futon, CouchDB will force an update to the view index and
will block Futon on trying to browse that design document. This makes it
almost impossible to be able to browse the view results as the indexing may
never catch up with the writes.
A possible solution to this is to provide a UI option (Checkbox) in Futon which
would append the "stale='ok'" onto all Futon view requests allowing for
instantaneous browsing of the current state of the DB view indexes. This
option would be off by default (current behavior).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-533) When multiple futon views are open, pointing at different couchdb's running on different ports, the 'recent databases' section erroneously shows DB's across all futons.
[ https://issues.apache.org/jira/browse/COUCHDB-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Rempe updated COUCHDB-533: Component/s: (was: HTTP Interface) Administration Console > When multiple futon views are open, pointing at different couchdb's running > on different ports, the 'recent databases' section erroneously shows DB's > across all futons. > > > Key: COUCHDB-533 > URL: https://issues.apache.org/jira/browse/COUCHDB-533 > Project: CouchDB > Issue Type: Bug > Components: Administration Console > Affects Versions: 0.11 >Reporter: Glenn Rempe > > I am running a local CouchDB on localhost:5984. I am also running a > production CouchDB on a remote host that I am doing SSH port forwarding for > which I have mapped to localhost:5994. I am accessing their futon instances > via: > http://localhost:5984/_utils/status.html > http://localhost:5994/_utils/status.html > Viewing the futon for both shows all databases in the 'Recent Databases' > section, across both instances, in both futons. This is incorrect as each > futon should only show the subset of DB's that it actually contains. > e.g. > http://localhost:5984/_utils/status.html - should only show 'foo_dev' > http://localhost:5994/_utils/status.html - should only show 'foo_production' > Clicking on a database that doesn't actually exist results in two javascript > alerts. The first of which says 'An error occurred retrieving a list of all > documents: no_db_file' > Expected behavior: > Futon should only show databases that exist on the instance being monitored > in the 'Recent Databases' section. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-534) Futon's status page, when polling to show current status, is sometimes out of sync with server. Full page refresh can bring it back in sync.
Futon's status page, when polling to show current status, is sometimes out of sync with server. Full page refresh can bring it back in sync. - Key: COUCHDB-534 URL: https://issues.apache.org/jira/browse/COUCHDB-534 Project: CouchDB Issue Type: Bug Components: Administration Console Affects Versions: 0.11 Reporter: Glenn Rempe I have seen the 'status' page in Futon get out of sync with the tasks actually running on the server on several occasions when relying on the automatic polling feature which polls the server via background request every N seconds. Doing a full browser page refresh seems to resolve the issue. When I have seen this issue it shows, ever after period page refreshes, the same status items which seem frozen, leading the user to believe that the processes of interest may be frozen. This is likely erroneous. So its seems there is a problem with the API used to retrieve background ajax info. Expected behavior : The polling based status page, should always show exactly the same info as the status page after a full page refresh. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-533) When multiple futon views are open, pointing at different couchdb's running on different ports, the 'recent databases' section erroneously shows DB's across all futons.
When multiple futon views are open, pointing at different couchdb's running on
different ports, the 'recent databases' section erroneously shows DB's across
all futons.
Key: COUCHDB-533
URL: https://issues.apache.org/jira/browse/COUCHDB-533
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 0.11
Reporter: Glenn Rempe
I am running a local CouchDB on localhost:5984. I am also running a production
CouchDB on a remote host that I am doing SSH port forwarding for which I have
mapped to localhost:5994. I am accessing their futon instances via:
http://localhost:5984/_utils/status.html
http://localhost:5994/_utils/status.html
Viewing the futon for both shows all databases in the 'Recent Databases'
section, across both instances, in both futons. This is incorrect as each
futon should only show the subset of DB's that it actually contains.
e.g.
http://localhost:5984/_utils/status.html - should only show 'foo_dev'
http://localhost:5994/_utils/status.html - should only show 'foo_production'
Clicking on a database that doesn't actually exist results in two javascript
alerts. The first of which says 'An error occurred retrieving a list of all
documents: no_db_file'
Expected behavior:
Futon should only show databases that exist on the instance being monitored in
the 'Recent Databases' section.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-532) Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
[ https://issues.apache.org/jira/browse/COUCHDB-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766938#action_12766938 ] Glenn Rempe commented on COUCHDB-532: - Jan, This would, in my opinion, be an excellent new facility to have. Based on your description it would indeed solve the issue at hand nicely if it were time and event based. Allowing for cron-like control over all of the major aspects of CouchDB would be powerful. Would you say this is a long term feature plan that is early in its design, or one that is close to reaching fruition? If its a longer term feature plan, perhaps the enhancement to do background indexing on stale=ok could be implemented as a stop-gap measure in the interim if its easy to add (and just as easy to remove later)? Speaking for myself I could use this feature yesterday. :-) > Have requests for views that use 'stale=ok' return whats in the index > immediately, but still kickoff background indexing process. > - > > Key: COUCHDB-532 > URL: https://issues.apache.org/jira/browse/COUCHDB-532 > Project: CouchDB > Issue Type: Improvement > Components: Database Core > Environment: All >Reporter: Glenn Rempe > > Currently a common use case for querying indexes from external applications > is to pass in the optional parameter 'stale=ok' which instructs the view to > return what is has in its index immediately without waiting for the index to > first catch up. > This is a *very* important feature as it allows applications to eliminate the > poor user experience of potential HTTP client timeouts or long'ish waits for > a query to return while the index is updated. > However, applications that make extensive (or in my case exlusive) use of > this feature are now essentially *required* to build some kind of script > which will do the indexing in the background since no user requests will > *ever* initiate indexing. This would be for example a cron script that makes > an HTTP call to the view every N hours. This becomes an operational issue as > you now have to write, maintain and schedule an external script which calls > each view in each of your DBs periodically. > I propose to the team, after some discussion with Paul Joseph Davis, that > view requests with the 'stale=ok' param continue to return results in the > as-is index to users immediately, but *also* kick off the background indexing > process. This would have the effect of removing the need for any external > maintenance scripts (aside from DB/View compaction which is a related but > separate issue). > In some use cases it might be true that indexing *must* occur outside of > 'normal business hours' due to processing load. If this is a requirement the > behavior of view indexing on 'stale' requests could be configurable (e.g. as > a boolean to always/never do this additional background indexing, or force BG > indexing every N requests/minutes using 'stale'). > Cheers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-532) Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process.
Have requests for views that use 'stale=ok' return whats in the index immediately, but still kickoff background indexing process. - Key: COUCHDB-532 URL: https://issues.apache.org/jira/browse/COUCHDB-532 Project: CouchDB Issue Type: Improvement Components: Database Core Environment: All Reporter: Glenn Rempe Currently a common use case for querying indexes from external applications is to pass in the optional parameter 'stale=ok' which instructs the view to return what is has in its index immediately without waiting for the index to first catch up. This is a *very* important feature as it allows applications to eliminate the poor user experience of potential HTTP client timeouts or long'ish waits for a query to return while the index is updated. However, applications that make extensive (or in my case exlusive) use of this feature are now essentially *required* to build some kind of script which will do the indexing in the background since no user requests will *ever* initiate indexing. This would be for example a cron script that makes an HTTP call to the view every N hours. This becomes an operational issue as you now have to write, maintain and schedule an external script which calls each view in each of your DBs periodically. I propose to the team, after some discussion with Paul Joseph Davis, that view requests with the 'stale=ok' param continue to return results in the as-is index to users immediately, but *also* kick off the background indexing process. This would have the effect of removing the need for any external maintenance scripts (aside from DB/View compaction which is a related but separate issue). In some use cases it might be true that indexing *must* occur outside of 'normal business hours' due to processing load. If this is a requirement the behavior of view indexing on 'stale' requests could be configurable (e.g. as a boolean to always/never do this additional background indexing, or force BG indexing every N requests/minutes using 'stale'). Cheers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-527) add a check to see that the driver directory exists
add a check to see that the driver directory exists --- Key: COUCHDB-527 URL: https://issues.apache.org/jira/browse/COUCHDB-527 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Glenn Rempe Adding ticket as reminder per Paul Davis. Should be assigned to him. add a check to see that the driver directory exists 'Eh, $PREFIX/lib/couchdb-$(VERSION/priv/lib' and mention in couch_server_sup.erl just before couch_util:start_driver() -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-213) config test failure for ports 80 and 443
[
https://issues.apache.org/jira/browse/COUCHDB-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667049#action_12667049
]
Glenn Rempe commented on COUCHDB-213:
-
I have updated my couchdb, which sits behind a proxy, and I am still seeing the
same behavior.
http://couchdb.rempe.org/_utils/couch_tests.html?script/couch_tests.js
This instance was built from source with commit ID tr...@737346
I can give you admin access for this instance. Please contact me by email and
I'll give you a password if you'd like to use it for testing.
I am still seeing the error message:
# Assertion failed: config.httpd.port == port
And I have confirmed that the new test is the one that is running. I ran this
in FireFox (latest) on Mac OS X. I cleared cache before running.
Perhaps this line is not evaluating correctly? In which case I think the
behavior would be identical to the old (broken) behavior.
if(server_port.length == 1 && CouchDB.inBrowser) {
Please re-open.
Thanks.
> config test failure for ports 80 and 443
>
>
> Key: COUCHDB-213
> URL: https://issues.apache.org/jira/browse/COUCHDB-213
> Project: CouchDB
> Issue Type: Bug
> Components: Test Suite
>Reporter: Dean Landolt
>Priority: Minor
>
> In the config test this assertion:
> T(config.httpd.port == port);
> depends on the line:
> var port = CouchDB.host.split(":").pop();
> But there's no :port on when running on 80 and 443. If config.httpd.port ==
> 80 or 443 the port assertion should be skipped (and perhaps the protocol path
> part checked for http:// or https://)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-219) Futon JS test for 'config' fails when CouchDB is served behind reverse proxy on default port (80)
Futon JS test for 'config' fails when CouchDB is served behind reverse proxy on
default port (80)
-
Key: COUCHDB-219
URL: https://issues.apache.org/jira/browse/COUCHDB-219
Project: CouchDB
Issue Type: Bug
Components: Test Suite
Environment: Apache reverse proxy in front of CouchDB
Reporter: Glenn Rempe
Priority: Minor
The config test in the CouchDB JS test suite has a hardcoded assumption that
the server port will always be available for use in the test. However, when
CDB is hosted behind a reverse proxy running on default port 80 there will be
no such port that is part of the URL.
Although not tested I assume this will also fail when behind an https (TLS)
proxy URL.
In this case the test fails showing:
Assertion failed: config.httpd.port == port
The offending line in this test is:
var port = CouchDB.host.split(":").pop();
Suggested fix would be to test whether the URL contains a ':'. If not, then
port should be set to 80 or 443 depending on whether the url is an http/https
url.
All other tests pass when behind a properly configured proxy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
