[jira] [Commented] (COUCHDB-1259) Replication ID is not stable if local server has a dynamic port number
[
https://issues.apache.org/jira/browse/COUCHDB-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090680#comment-13090680
]
Jason Smith commented on COUCHDB-1259:
--
Hi, Jens. Yes, I take your point to stick to the specific topic. Thanks.
To summarize my long rants: A Couch UUID already exists. A database UUID won't
work. Allowing the client to provide the replication ID would be great!
> Replication ID is not stable if local server has a dynamic port number
> --
>
> Key: COUCHDB-1259
> URL: https://issues.apache.org/jira/browse/COUCHDB-1259
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1
>Reporter: Jens Alfke
>
> I noticed that when Couchbase Mobile running on iOS replicates to/from a
> remote server (on iriscouch in this case), the replication has to fetch the
> full _changes feed every time it starts. Filipe helped me track down the
> problem -- the replication ID is coming out different every time. The reason
> for this is that the local port number, which is one of the inputs to the
> hash that generates the replication ID, is randomly assigned by the OS. (I.e.
> it uses a port number of 0 when opening its listener socket.) This is because
> there could be multiple apps using Couchbase Mobile running on the same
> device and we can't have their ports colliding.
> The underlying problem is that CouchDB is attempting to generate a unique ID
> for a particular pair of {source, destination} databases, but it's basing it
> on attributes that aren't fundamental to the database and can change, like
> the hostname or port number.
> One solution, proposed by Filipe and me, is to assign each database (or each
> server?) a random UUID when it's created, and use that to generate
> replication IDs.
> Another solution, proposed by Damien, is to have CouchDB let the client work
> out the replication ID on its own, and set it as a property in the
> replication document (or the JSON body of a _replicate request.) This is even
> more flexible and will handle tricky scenarios like full P2P replication
> where there may be no low-level way to uniquely identify the remote database
> being synced with.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090592#comment-13090592
]
Filipe Manana commented on COUCHDB-1258:
Robert,
That solution works perfectly fine for my case. Memory consumption doesn't go
beyond ~40Mb (for the whole Erlang VM). Perhaps a lower limit (to match the
work queue max items) could be generally better, I haven't tested it however.
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090551#comment-13090551
]
Paul Joseph Davis commented on COUCHDB-1257:
Totally just realized this is the same issue as the mailing list. Either way,
yeah...
> Reduction detection in share/server/views.js runReduce to primitive for some
> use cases and overall reduce architecture
> --
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
>Affects Versions: 1.1
> Environment: Architecture independent
>Reporter: Chris Stockton
>Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to
> the protocol in which couchdb actually talks to the view server
> (server/main.js), the second part as discussed here [1] in short is the
> growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a
> reduce function for every doc called, a separate read line is delegated a
> reset call. I do not know the internals of couchdb but it seems like it could
> be a performance improvement if on a rereduce call, as it appears in the
> server Loop function every time reduce or rereduce is called the state is
> first reset with "reset", to make a minor re-factor and reset the state
> within the Views dispatcher itself. This saves some likely blocking io back
> and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection
> done, I think that it should be changed to require a sampling of data, the
> easiest fix is to simply change (the very arbitrary) [2] into a large length,
> such as * 10 or something, but the upper bounds will always be reached for
> some data set. A new algorithm could take in consideration the total reduce
> calls thus far, as well as the initial size of the very first reduce call, to
> see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count
> of rereduce calls thus far, or give some kind of handle where this logic
> could be reliably kept track of inside of main.js, as it stands now your only
> hope would be to use the reduce function itself as a lookup, which is not
> safe under all edge cases. With this information you can keep the initial
> rereduce calls result size, and the total number of calls, to make sure that
> aside from initial aggregation or structure initialization the structure is
> remaining constant.
> There would be several other routes to go to fix this, but I think it is
> valid to be considered. I could volunteer my time to write a patch if a
> consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) >
> State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
That link is to trunk. We should backport it to 1.1.x so it's in 1.1.1 as well. Perhaps even a 1.0.4. On Wed, Aug 24, 2011 at 4:05 PM, Chris Stockton wrote: > Hello, > > On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis > wrote: >> I bet you're hitting a bug we just recently fixed on trunk. Basically, >> there was a possibility that errors in some of the JS functions would >> end up causing a couchjs process to be come unusable without removing >> it from the list. Eventually there wouldn't be any spots left and >> clients would get timeouts like you see. >> >> Patch is at [1]. If it doesn't apply cleanly, you really only need the >> bits from couch_os_process.erl and couch_query_servers.erl. The rest >> is just test code. >> >> https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9 >> > > Right after sending this I finally saw what the issue was, our bug > report here: > https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484 > as a side effect was leaving lingering processes ultimately leading to > instability of couch. > > We are working to patch our reduce issue, and will look at applying > that commit perhaps once it hits mainline couch? > > Thanks for the quick response, > > -Chris >
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090548#comment-13090548
]
Paul Joseph Davis commented on COUCHDB-1257:
We just fixed this in trunk. It's a small patch to fix the issue if you're
interested. You only need the couch_os_file.erl and couch_query_servers.erl
diffs to make it work.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
> Reduction detection in share/server/views.js runReduce to primitive for some
> use cases and overall reduce architecture
> --
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
>Affects Versions: 1.1
> Environment: Architecture independent
>Reporter: Chris Stockton
>Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to
> the protocol in which couchdb actually talks to the view server
> (server/main.js), the second part as discussed here [1] in short is the
> growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a
> reduce function for every doc called, a separate read line is delegated a
> reset call. I do not know the internals of couchdb but it seems like it could
> be a performance improvement if on a rereduce call, as it appears in the
> server Loop function every time reduce or rereduce is called the state is
> first reset with "reset", to make a minor re-factor and reset the state
> within the Views dispatcher itself. This saves some likely blocking io back
> and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection
> done, I think that it should be changed to require a sampling of data, the
> easiest fix is to simply change (the very arbitrary) [2] into a large length,
> such as * 10 or something, but the upper bounds will always be reached for
> some data set. A new algorithm could take in consideration the total reduce
> calls thus far, as well as the initial size of the very first reduce call, to
> see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count
> of rereduce calls thus far, or give some kind of handle where this logic
> could be reliably kept track of inside of main.js, as it stands now your only
> hope would be to use the reduce function itself as a lookup, which is not
> safe under all edge cases. With this information you can keep the initial
> rereduce calls result size, and the total number of calls, to make sure that
> aside from initial aggregation or structure initialization the structure is
> remaining constant.
> There would be several other routes to go to fix this, but I think it is
> valid to be considered. I could volunteer my time to write a patch if a
> consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) >
> State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-1232) trunk does not build with spidermonkey 1.7
[ https://issues.apache.org/jira/browse/COUCHDB-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds reassigned COUCHDB-1232: -- Assignee: Paul Joseph Davis (was: Randall Leeds) Paul said he's working on this right now. > trunk does not build with spidermonkey 1.7 > -- > > Key: COUCHDB-1232 > URL: https://issues.apache.org/jira/browse/COUCHDB-1232 > Project: CouchDB > Issue Type: Improvement > Components: Build System >Affects Versions: 1.2 > Environment: Gentoo Linux >Reporter: James Marca >Assignee: Paul Joseph Davis >Priority: Minor > Fix For: 1.2 > > Original Estimate: 4h > Remaining Estimate: 4h > > Original bug report: "configure does not require spidermonkey 1.8.5, but > trunk (1.2) failed to build without it." > Trunk should fall back to a slightly older spidermonkey and configure > definitely fails if you don't have either. > However, by popular demand I'm told I should make trunk compile with 1.7 > again, so I will. I'm changing this ticket to track that. > Thanks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-752) InternalError: script stack space quota is exhausted
[
https://issues.apache.org/jira/browse/COUCHDB-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds reassigned COUCHDB-752:
-
Assignee: Randall Leeds
> InternalError: script stack space quota is exhausted
>
>
> Key: COUCHDB-752
> URL: https://issues.apache.org/jira/browse/COUCHDB-752
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
>Affects Versions: 0.11
>Reporter: Benoit Chesneau
>Assignee: Randall Leeds
>Priority: Critical
> Fix For: 1.2
>
> Attachments: ddoc.patch
>
>
> Couchdb freeze and use full CPU after accessing to a 93MB designdoc:
> debug] [<0.91.0>] Spawning new group server for view group _design/aimpl in
> database aimpl.
> [debug] [<0.94.0>] request_group {Pid, Seq} {<0.116.0>,61}
> [debug] [<0.81.0>] New task status for aimpl _design/aimpl: Finishing.
> [debug] [<0.126.0>] OS Process Start :: #Port<0.2040>
> [debug] [<0.87.0>] Teach ddoc to new proc {proc,<0.126.0>,<<"javascript">>,[],
> {couch_os_process,prompt},
> {couch_os_process,set_timeout},
> {couch_os_process,stop}} with DDocKey:
> {<<"_design/aimpl">>,
>
> <<"30-2ab623a62b83b0d9f2616ee209f62659">>}
> InternalError: script stack space quota is exhausted
> Most of data are in attachments :
> enlil:couchapp benoitc$ du -sh aimpl
> 93M aimpl
> enlil:couchapp benoitc$ du -sh aimpl/_attachments/
> 93M aimpl/_attachments/
> about 100k in views/shows/...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-1012) Utility to help plugin developers manage paths
[ https://issues.apache.org/jira/browse/COUCHDB-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds reassigned COUCHDB-1012: -- Assignee: Benoit Chesneau (was: Randall Leeds) > Utility to help plugin developers manage paths > -- > > Key: COUCHDB-1012 > URL: https://issues.apache.org/jira/browse/COUCHDB-1012 > Project: CouchDB > Issue Type: New Feature > Components: Build System >Reporter: Randall Leeds >Assignee: Benoit Chesneau > Fix For: 1.2 > > Attachments: > 0001-add-couch-config-file-used-to-ease-the-build-of-plug.patch, > 0001-add-couch-config-file-used-to-ease-the-build-of-plug.patch, > 0001-support-pkg-config-for-plugins-COUCHDB-1012.patch > > > Developers may want to write plugins (like GeoCouch) for CouchDB. Many hooks > in the configuration system allow loading arbitrary Erlang modules to handle > various internal tasks, but currently there is no straightforward and > portable way for developers of these plugins to discover the location of the > CouchDB library files. > Two options that have been proposed are to use pkg-config or install a > separate script that could be invoked (e.g. as couch-config --erl-libs) to > discover important CouchDB installation paths. > As far as I know the loudest argument against pkg-config is lack of support > for Windows. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
Hello, On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis wrote: > I bet you're hitting a bug we just recently fixed on trunk. Basically, > there was a possibility that errors in some of the JS functions would > end up causing a couchjs process to be come unusable without removing > it from the list. Eventually there wouldn't be any spots left and > clients would get timeouts like you see. > > Patch is at [1]. If it doesn't apply cleanly, you really only need the > bits from couch_os_process.erl and couch_query_servers.erl. The rest > is just test code. > > https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9 > Right after sending this I finally saw what the issue was, our bug report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484 as a side effect was leaving lingering processes ultimately leading to instability of couch. We are working to patch our reduce issue, and will look at applying that commit perhaps once it hits mainline couch? Thanks for the quick response, -Chris
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090484#comment-13090484
]
Chris Stockton commented on COUCHDB-1257:
-
Adam it appears this error is a bit more serious then original understood.
While this has been happening we were having a similar error with views no
longer returning results and just timing out. It seems each reduce over flow
error results in a couchjs process remaining open forever.
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
14
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
24
This ultimately leads to couch becoming unstable.
> Reduction detection in share/server/views.js runReduce to primitive for some
> use cases and overall reduce architecture
> --
>
> Key: COUCHDB-1257
> URL: https://issues.apache.org/jira/browse/COUCHDB-1257
> Project: CouchDB
> Issue Type: Bug
> Components: JavaScript View Server
>Affects Versions: 1.1
> Environment: Architecture independent
>Reporter: Chris Stockton
>Priority: Minor
> Labels: javascript, patch
>
> There is two parts to this report, one is my question I have in regards to
> the protocol in which couchdb actually talks to the view server
> (server/main.js), the second part as discussed here [1] in short is the
> growth detection [2] done in share/server/views.js.
> First issue:
> While troubleshooting and tracking down this issue, I saw that when running a
> reduce function for every doc called, a separate read line is delegated a
> reset call. I do not know the internals of couchdb but it seems like it could
> be a performance improvement if on a rereduce call, as it appears in the
> server Loop function every time reduce or rereduce is called the state is
> first reset with "reset", to make a minor re-factor and reset the state
> within the Views dispatcher itself. This saves some likely blocking io back
> and forth.
> Reduction detection:
> The portion of this issue that actually has affected us is the detection
> done, I think that it should be changed to require a sampling of data, the
> easiest fix is to simply change (the very arbitrary) [2] into a large length,
> such as * 10 or something, but the upper bounds will always be reached for
> some data set. A new algorithm could take in consideration the total reduce
> calls thus far, as well as the initial size of the very first reduce call, to
> see if a large aggregation of some sort is being created.
> Suggested change:
> Change the signature of rereduce call to main.js to include the total count
> of rereduce calls thus far, or give some kind of handle where this logic
> could be reliably kept track of inside of main.js, as it stands now your only
> hope would be to use the reduce function itself as a lookup, which is not
> safe under all edge cases. With this information you can keep the initial
> rereduce calls result size, and the total number of calls, to make sure that
> aside from initial aggregation or structure initialization the structure is
> remaining constant.
> There would be several other routes to go to fix this, but I think it is
> valid to be considered. I could volunteer my time to write a patch if a
> consensus could be reached on behavior, if not denied all together.
> -Chris
> [1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
> [2] if (State.query_config && State.query_config.reduce_limit &&
> [2] reduce_length > 200 && ((reduce_length * 2) >
> State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090478#comment-13090478
]
Robert Newson commented on COUCHDB-1258:
you could try this to see if it helps with memory usage. Verified locally that
it radically reduces the number of checkpoints.
diff --git a/src/couchdb/couch_view_group.erl b/src/couchdb/couch_view_group.erl
index cde949b..7002d07 100644
--- a/src/couchdb/couch_view_group.erl
+++ b/src/couchdb/couch_view_group.erl
@@ -243,7 +243,7 @@ handle_cast({partial_update, Pid, NewGroup},
#group_state{updater_pid=Pid}
group = Group
} = State,
NewSeq = NewGroup#group.current_seq,
-case NewSeq > Group#group.current_seq of
+case NewSeq > 1000 + Group#group.current_seq of
true ->
?LOG_INFO("checkpointing view update at seq ~p for ~s ~s", [NewSeq,
DbName, NewGroup#group.name]),
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
I bet you're hitting a bug we just recently fixed on trunk. Basically,
there was a possibility that errors in some of the JS functions would
end up causing a couchjs process to be come unusable without removing
it from the list. Eventually there wouldn't be any spots left and
clients would get timeouts like you see.
Patch is at [1]. If it doesn't apply cleanly, you really only need the
bits from couch_os_process.erl and couch_query_servers.erl. The rest
is just test code.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
On Wed, Aug 24, 2011 at 3:42 PM, Chris Stockton
wrote:
> We have about 3000 active databases on our server, about 200 of those
> users (most active ones) keep randomly failing to load views. I am
> having difficulty trying to decipher this stack trace to make any
> sense of it, does anyone have any suggestions about what is happening
> here? Are we just reaching some sort of capacity limit? The machines
> are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
> are fine on disk space, and requests per second to be honest is around
> 1 with plenty of idle time between little spurts.
>
> -Chris
>
>
>
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] 'GET'
> /db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
> {1,
>
> 1} from "WITHHELD"
> Headers: [{'Accept',"*/*"},
> {'Authorization',"Basic WITHHELD"},
> {'Content-Type',"application/json; charset=utf-8"},
> {'Host',"WITHHELD:5984"},
> {'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] OAuth Params:
> [{"skip","0"},
> {"key",
> "[\"model\",\"M4e551a78a9b3d\"]"},
> {"limit","25"}]
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] request_group
> {Pid, Seq} {<0.25836.0>,1077}
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
> db_946 _design/views: Processed 0 of 25 changes (0%)
> [Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
> db_946 _design/views: Finishing.
> [Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] 'GET'
> /_all_dbs {1,1} from "WITHHELD"
> Headers: [{'Accept',"*/*"},
> {'Authorization',"Basic WITHHELD"},
> {'Content-Type',"application/json; charset=utf-8"},
> {'Host',"WITHHELD:5984"},
> {'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
> [Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] OAuth Params: []
> [Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.25836.0>] Exit from linked
> pid: {<0.27488.217>,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,
> <<"javascript">>}]}}}
> [Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] ** Generic
> server <0.25836.0> terminating
> ** Last message in was {'EXIT',<0.27488.217>,
> {timeout,
> {gen_server,call,
> [couch_query_servers,
> {get_proc,<<"javascript">>}]}}}
> ** When Server state == {group_state,undefined,<<"db_946">>,
> {"/WITHHELD/5984/views",<<"db_946">>,
> {group,
> <<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
> 170,203>>,
> nil,nil,<<"_design/views">>,<<"javascript">>,[],
> [{view,0,0,0,
> [<<"getByType">>],
> <<"function(doc) {\n \n var sdoc =
> eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
> sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
> sdoc.data);\n } else if((typeof doc.type)
> !== \"undefined\" && doc.type === \"couchimport\") {\n
> emit([doc.type, doc.time], null);\n }\n}">>,
> nil,[],[]}],
> {[]},
> nil,0,0,nil,nil}},
> {group,
> <<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
> 170,203>>,
> {db,<0.6703.0>,<0.6717.0>,nil,
> <<"1313769124135037">>,<0.24135.71>,<0.24137.71>,
> {db_header,5,1077,0,
> {1011747,{410,120}},
> {1014120,530},
> {1022521,[]},
> 0,nil,nil,1000},
> 1077,
> {btree,<0
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090466#comment-13090466
]
Paul Joseph Davis commented on COUCHDB-1258:
Sounds reasonable. A map function that emits no keys coupled with a lot of
deleted documents means that the partial_update frequency will be quite large.
couch_work_queue's behave quite poorly when the dequeue side is faster than the
queue side and end up basically not bulking updates together. A quick test
would be to guard the partial_update message to only fire at most every 1000
writes. A quick hack to store calls in the process dict in do_writes should
give a quick idea if it's the partial updates.
A should_flush for every partial update seems a bit extreme. Although I can't
think of a better solution that doesn't just make an equivalent to should_flush
that's more specific to this case.
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Our views keep failing with timeouts until a couchdb restart
We have about 3000 active databases on our server, about 200 of those
users (most active ones) keep randomly failing to load views. I am
having difficulty trying to decipher this stack trace to make any
sense of it, does anyone have any suggestions about what is happening
here? Are we just reaching some sort of capacity limit? The machines
are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
are fine on disk space, and requests per second to be honest is around
1 with plenty of idle time between little spurts.
-Chris
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] 'GET'
/db_946/_design/views/_view/getByType?skip=0&key=["model","M4e551a78a9b3d"]&limit=25
{1,
1} from "WITHHELD"
Headers: [{'Accept',"*/*"},
{'Authorization',"Basic WITHHELD"},
{'Content-Type',"application/json; charset=utf-8"},
{'Host',"WITHHELD:5984"},
{'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] OAuth Params:
[{"skip","0"},
{"key",
"[\"model\",\"M4e551a78a9b3d\"]"},
{"limit","25"}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.14283.217>] request_group
{Pid, Seq} {<0.25836.0>,1077}
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
db_946 _design/views: Processed 0 of 25 changes (0%)
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [<0.82.0>] New task status for
db_946 _design/views: Finishing.
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] 'GET'
/_all_dbs {1,1} from "WITHHELD"
Headers: [{'Accept',"*/*"},
{'Authorization',"Basic WITHHELD"},
{'Content-Type',"application/json; charset=utf-8"},
{'Host',"WITHHELD:5984"},
{'User-Agent',"PECL::HTTP/1.7.0 (PHP/5.3.4)"}]
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [<0.31520.215>] OAuth Params: []
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [<0.25836.0>] Exit from linked
pid: {<0.27488.217>,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,
<<"javascript">>}]}}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [<0.25836.0>] ** Generic
server <0.25836.0> terminating
** Last message in was {'EXIT',<0.27488.217>,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,<<"javascript">>}]}}}
** When Server state == {group_state,undefined,<<"db_946">>,
{"/WITHHELD/5984/views",<<"db_946">>,
{group,
<<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203>>,
nil,nil,<<"_design/views">>,<<"javascript">>,[],
[{view,0,0,0,
[<<"getByType">>],
<<"function(doc) {\n \n var sdoc =
eval(uneval(doc));\n if((typeof doc.model) !== \"undefined\") {\n
sdoc.data.rev = doc._rev;\nemit([doc.type, doc.model],
sdoc.data);\n } else if((typeof doc.type)
!== \"undefined\" && doc.type === \"couchimport\") {\n
emit([doc.type, doc.time], null);\n }\n}">>,
nil,[],[]}],
{[]},
nil,0,0,nil,nil}},
{group,
<<88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203>>,
{db,<0.6703.0>,<0.6717.0>,nil,
<<"1313769124135037">>,<0.24135.71>,<0.24137.71>,
{db_header,5,1077,0,
{1011747,{410,120}},
{1014120,530},
{1022521,[]},
0,nil,nil,1000},
1077,
{btree,<0.24135.71>,
{1011747,{410,120}},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,<0.24135.71>,
{1014120,530},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,<0.24135.71>,
{1022521,[]},
#Fun,
#Fun,
#Fun,nil},
1077,<<"db_946">>,
"/WITHHELD/5984/dbs/db_946.couch",[],[],
[jira] [Commented] (COUCHDB-1238) CouchDB uses _users db for storing oauth credentials
[ https://issues.apache.org/jira/browse/COUCHDB-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090431#comment-13090431 ] Pete Vander Giessen commented on COUCHDB-1238: -- @Filipe (just realized that I misspelled your name twice in my last comment; fixed now -- sorry): you are the expert. Any changes that you deem necessary are fine by me :-) (And reading through the code, it looks like all the changes you made make things better.) Let me know if I need to do anything else on this one -- happy to help in any way I can. > CouchDB uses _users db for storing oauth credentials > > > Key: COUCHDB-1238 > URL: https://issues.apache.org/jira/browse/COUCHDB-1238 > Project: CouchDB > Issue Type: New Feature > Components: Database Core >Affects Versions: 1.1 >Reporter: Pete Vander Giessen >Assignee: Filipe Manana > Fix For: 1.2 > > Attachments: git_commits_as_patch.zip, oauth_users_db_patch.zip > > > We want to store oauth credentials in the _users db, rather than in the .ini. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090410#comment-13090410
]
Filipe Manana commented on COUCHDB-1258:
I have a similar case. Database with about 2.6 million deleted documents.
The map function is not emitting any key/value and memory consumption goes to
about 4Gb.
After the indexer finishes, and the system goes idle, memory usage stays at
about 2Gb forever.
Inspecting the erlang processes, a couch_view_group process is using about 2Gb
of memory:
7> process_info(list_to_pid("<0.276.0>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.278.0>,<0.132.0>]},
{dictionary,[{'$ancestors',[<0.275.0>]},
{'$initial_call',{couch_view_group,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.37.0>},
{total_heap_size,323345205},
{heap_size,38263080},
{stack_size,9},
{reductions,1041902893},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,538}]},
{suspending,[]}]
8>
Manually invoking the garbage collector against this process releases memory
and after that the process is using only about 7Kb of memory:
8> erlang:garbage_collect(list_to_pid("<0.276.0>")).
true
9> process_info(list_to_pid("<0.276.0>")).
[{current_function,{gen_server,loop,6}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.278.0>,<0.132.0>]},
{dictionary,[{'$ancestors',[<0.275.0>]},
{'$initial_call',{couch_view_group,init,1}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.37.0>},
{total_heap_size,987},
{heap_size,987},
{stack_size,9},
{reductions,1041902893},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]
10>
I have no idea why the view group is using so much memory.
My best call is that we need to call couch_util:should_flush/1 everytime we
receive a partial_update message in the view group.
This was with OT R14B03.
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see:
[jira] [Commented] (COUCHDB-1259) Replication ID is not stable if local server has a dynamic port number
[
https://issues.apache.org/jira/browse/COUCHDB-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090396#comment-13090396
]
Jens Alfke commented on COUCHDB-1259:
-
@Jason — I see where you're coming from with "bad web citizen" but it refers
only to a specific usage of CouchDB as a traditional web server. We have
equally valid uses of it that do not act as traditional servers nor have stable
URLs, for which we absolutely need this fix. I don't think arguments about
whether URLs change or not are productive here in a specific bug report.
> Replication ID is not stable if local server has a dynamic port number
> --
>
> Key: COUCHDB-1259
> URL: https://issues.apache.org/jira/browse/COUCHDB-1259
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.1
>Reporter: Jens Alfke
>
> I noticed that when Couchbase Mobile running on iOS replicates to/from a
> remote server (on iriscouch in this case), the replication has to fetch the
> full _changes feed every time it starts. Filipe helped me track down the
> problem -- the replication ID is coming out different every time. The reason
> for this is that the local port number, which is one of the inputs to the
> hash that generates the replication ID, is randomly assigned by the OS. (I.e.
> it uses a port number of 0 when opening its listener socket.) This is because
> there could be multiple apps using Couchbase Mobile running on the same
> device and we can't have their ports colliding.
> The underlying problem is that CouchDB is attempting to generate a unique ID
> for a particular pair of {source, destination} databases, but it's basing it
> on attributes that aren't fundamental to the database and can change, like
> the hostname or port number.
> One solution, proposed by Filipe and me, is to assign each database (or each
> server?) a random UUID when it's created, and use that to generate
> replication IDs.
> Another solution, proposed by Damien, is to have CouchDB let the client work
> out the replication ID on its own, and set it as a property in the
> replication document (or the JSON body of a _replicate request.) This is even
> more flexible and will handle tricky scenarios like full P2P replication
> where there may be no low-level way to uniquely identify the remote database
> being synced with.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090391#comment-13090391 ] Robert Newson commented on COUCHDB-431: --- Don't get me wrong, the lines in this module are far too long anyway, but reindenting makes it harder to see the substantive change you're making. The first one where you added "OPTIONS" was the one I was referring to, but I had failed to notice that the before/after were different. Given the alternative is to make these overlong lines even longer, I guess there's no need to change it, but I'd refactor this into two commits, one which simply wraps these long lines better and a second that then adds the extra bits. Not a deal breaker for me if you don't do it. > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, > 0001-cors-support.-should-fix-COUCHDB-431.patch, > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, > test_cors2.tgz > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Chesneau updated COUCHDB-431: Attachment: 0001-cors-support.-should-fix-COUCHDB-431.patch new version of the patch. removing spurious log & checkout back run.tpl . > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, > 0001-cors-support.-should-fix-COUCHDB-431.patch, > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, > test_cors2.tgz > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090384#comment-13090384 ] Benoit Chesneau commented on COUCHDB-431: - mmm by indentation change do you mean the new lines added with new options ? I can probably put them back on one line if it's better. Or did I miss other things (which is totally possible I'm a little blind tonight)? New patch remove this spurious log is coming. > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, > test_cors2.tgz > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090381#comment-13090381 ] Robert Newson commented on COUCHDB-431: --- The new patch combines indentation changes with code changes, can that be separated? > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, > test_cors2.tgz > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[
https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090382#comment-13090382
]
Robert Newson commented on COUCHDB-431:
---
also '?LOG_INFO("la", [])' probably shouldn't be in here, right? :)
> Support cross domain XMLHttpRequest (XHR) calls by implementing Access
> Control spec
> ---
>
> Key: COUCHDB-431
> URL: https://issues.apache.org/jira/browse/COUCHDB-431
> Project: CouchDB
> Issue Type: New Feature
> Components: HTTP Interface
>Affects Versions: 0.9
>Reporter: James Burke
>Assignee: Benoit Chesneau
>Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch,
> A_0001-Generalize-computing-the-appropriate-headers-for-any.patch,
> A_0002-Send-server-headers-for-externals-responses.patch,
> A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch,
> A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html,
> test_cors2.tgz
>
>
> Historically, browsers have been restricted to making XMLHttpRequests (XHRs)
> to the same origin (domain) as the web page making the request. However, the
> latest browsers now support cross-domain requests by implementing the Access
> Control spec from the W3C:
> http://dev.w3.org/2006/waf/access-control/
> In order to keep older servers safe that assume browsers only do same-domain
> requests, the Access Control spec requires the server to opt-in to allow
> cross domain requests by the use of special HTTP headers and supporting some
> "pre-flight" HTTP calls.
> Why should CouchDB support this: in larger, high traffic site, it is common
> to serve the static UI files from a separate, differently scaled server
> complex than the data access/API server layer. Also, there are some API
> services that are meant to be centrally hosted, but allow API consumers to
> use the API from different domains. In these cases, the UI in the browser
> would need to do cross domain requests to access CouchDB servers that act as
> the API/data access server layer.
> JSONP is not enough in these cases since it is limited to GET requests, so no
> POSTing or PUTing of documents.
> Some information from Firefox's perspective (functionality available as of
> Firefox 3.5):
> https://developer.mozilla.org/en/HTTP_access_control
> And information on Safari/Webkit (functionality in latest WebKit and Safari
> 4):
> http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html
> IE 8 also uses the Access Control spec, but the requests have to go through
> their XDomainRequest object (XDR):
> http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx
> and I thought IE8 only allowed GET or POST requests through their XDR.
> But as far as CouchDB is concerned, implementing the Access Control headers
> should be enough, and hopefully IE 9 will allow normal xdomain requests via
> XHR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[
https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Chesneau updated COUCHDB-431:
Attachment: test_cors2.tgz
0001-cors-support.-should-fix-COUCHDB-431.patch
This patch add support for CORS based on http://www.w3.org/TR/cors/ spec +
tests.
This ia a different implentation from the one proposed by Jason . It uses the
process registry to pass custom headers so we can manage authentication easily.
Supported :
- simple & preflight requests.
- possibility to forbid access per db by settings origins in secob. (see the
erlang tests) :
{
...
"origins": ["http://someorigin";]
}
- CORS headers are only available if origin header is present in the request
(like the spec say).
test_cors2.tgz is also attached to test it. To use it untar the file, push the
couchapp using couchapp or erica to a database named couch then open cors.html
file under another server or from the file system.
> Support cross domain XMLHttpRequest (XHR) calls by implementing Access
> Control spec
> ---
>
> Key: COUCHDB-431
> URL: https://issues.apache.org/jira/browse/COUCHDB-431
> Project: CouchDB
> Issue Type: New Feature
> Components: HTTP Interface
>Affects Versions: 0.9
>Reporter: James Burke
>Assignee: Benoit Chesneau
>Priority: Minor
> Fix For: 1.2
>
> Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch,
> A_0001-Generalize-computing-the-appropriate-headers-for-any.patch,
> A_0002-Send-server-headers-for-externals-responses.patch,
> A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch,
> A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html,
> test_cors2.tgz
>
>
> Historically, browsers have been restricted to making XMLHttpRequests (XHRs)
> to the same origin (domain) as the web page making the request. However, the
> latest browsers now support cross-domain requests by implementing the Access
> Control spec from the W3C:
> http://dev.w3.org/2006/waf/access-control/
> In order to keep older servers safe that assume browsers only do same-domain
> requests, the Access Control spec requires the server to opt-in to allow
> cross domain requests by the use of special HTTP headers and supporting some
> "pre-flight" HTTP calls.
> Why should CouchDB support this: in larger, high traffic site, it is common
> to serve the static UI files from a separate, differently scaled server
> complex than the data access/API server layer. Also, there are some API
> services that are meant to be centrally hosted, but allow API consumers to
> use the API from different domains. In these cases, the UI in the browser
> would need to do cross domain requests to access CouchDB servers that act as
> the API/data access server layer.
> JSONP is not enough in these cases since it is limited to GET requests, so no
> POSTing or PUTing of documents.
> Some information from Firefox's perspective (functionality available as of
> Firefox 3.5):
> https://developer.mozilla.org/en/HTTP_access_control
> And information on Safari/Webkit (functionality in latest WebKit and Safari
> 4):
> http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html
> IE 8 also uses the Access Control spec, but the requests have to go through
> their XDomainRequest object (XDR):
> http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx
> and I thought IE8 only allowed GET or POST requests through their XDR.
> But as far as CouchDB is concerned, implementing the Access Control headers
> should be enough, and hopefully IE 9 will allow normal xdomain requests via
> XHR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on COUCHDB-431 started by Benoit Chesneau. > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Chesneau reassigned COUCHDB-431: --- Assignee: Benoit Chesneau (was: Randall Leeds) > Support cross domain XMLHttpRequest (XHR) calls by implementing Access > Control spec > --- > > Key: COUCHDB-431 > URL: https://issues.apache.org/jira/browse/COUCHDB-431 > Project: CouchDB > Issue Type: New Feature > Components: HTTP Interface >Affects Versions: 0.9 >Reporter: James Burke >Assignee: Benoit Chesneau >Priority: Minor > Fix For: 1.2 > > Attachments: > A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, > A_0002-Send-server-headers-for-externals-responses.patch, > A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, > A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html > > > Historically, browsers have been restricted to making XMLHttpRequests (XHRs) > to the same origin (domain) as the web page making the request. However, the > latest browsers now support cross-domain requests by implementing the Access > Control spec from the W3C: > http://dev.w3.org/2006/waf/access-control/ > In order to keep older servers safe that assume browsers only do same-domain > requests, the Access Control spec requires the server to opt-in to allow > cross domain requests by the use of special HTTP headers and supporting some > "pre-flight" HTTP calls. > Why should CouchDB support this: in larger, high traffic site, it is common > to serve the static UI files from a separate, differently scaled server > complex than the data access/API server layer. Also, there are some API > services that are meant to be centrally hosted, but allow API consumers to > use the API from different domains. In these cases, the UI in the browser > would need to do cross domain requests to access CouchDB servers that act as > the API/data access server layer. > JSONP is not enough in these cases since it is limited to GET requests, so no > POSTing or PUTing of documents. > Some information from Firefox's perspective (functionality available as of > Firefox 3.5): > https://developer.mozilla.org/en/HTTP_access_control > And information on Safari/Webkit (functionality in latest WebKit and Safari > 4): > http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html > IE 8 also uses the Access Control spec, but the requests have to go through > their XDomainRequest object (XDR): > http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx > and I thought IE8 only allowed GET or POST requests through their XDR. > But as far as CouchDB is concerned, implementing the Access Control headers > should be enough, and hopefully IE 9 will allow normal xdomain requests via > XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: wrong case in Last-Modified
I bet they welcome patches too! :P On 24 Aug 2011, at 18:16, Andrey Somov wrote: >> >>> (but I would not mind if someone would change it to look perfect :) >> >> Patches welcome! >> > > I am sorry. I looked in the source and the header is generated by mochiweb.
Re: wrong case in Last-Modified
> > > (but I would not mind if someone would change it to look perfect :) > > Patches welcome! > I am sorry. I looked in the source and the header is generated by mochiweb.
Re: wrong case in Last-Modified
On 24 Aug 2011, at 17:49, Andrey Somov wrote: > (but I would not mind if someone would change it to look perfect :) Patches welcome!
Re: wrong case in Last-Modified
Thank you. (but I would not mind if someone would change it to look perfect :)
Re: wrong case in Last-Modified
HTTP headers are required to be treated case-insensitively (http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2). CouchDB supports http 1.0 and 1.1 but returns a 1.0 response to a 1.0 request. Using wget on my machine gives a 1.1 response, but perhaps your wget is older (it used to only support 1.0, iirc). B. On 24 August 2011 17:28, Andrey Somov wrote: > Hi all, > I have noticed that the HTTP header contains lower case value for > 'Last-Modified' > (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.29) > > ~$ wget -q -O - "$@"http://localhost:5984/favicon.ico --proxy=off > --server-response > HTTP/1.0 200 OK > Server: CouchDB/1.2.0a-1151390 (Erlang OTP/R13B03) > last-modified: Tue, 26 Jul 2011 04:19:47 GMT > Expires: Fri, 24 Aug 2012 15:02:33 GMT > Date: Wed, 24 Aug 2011 15:02:33 GMT > Content-Type: image/x-icon > Content-Length: 9326 > Cache-Control: public, max-age=31536000 > > By the way, the protocol in the reply is HTTP 1.0. Does it mean that CouchDB > does not support HTTP 1.1 ? > > - > Andrey >
wrong case in Last-Modified
Hi all, I have noticed that the HTTP header contains lower case value for 'Last-Modified' (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.29) ~$ wget -q -O - "$@"http://localhost:5984/favicon.ico --proxy=off --server-response HTTP/1.0 200 OK Server: CouchDB/1.2.0a-1151390 (Erlang OTP/R13B03) last-modified: Tue, 26 Jul 2011 04:19:47 GMT Expires: Fri, 24 Aug 2012 15:02:33 GMT Date: Wed, 24 Aug 2011 15:02:33 GMT Content-Type: image/x-icon Content-Length: 9326 Cache-Control: public, max-age=31536000 By the way, the protocol in the reply is HTTP 1.0. Does it mean that CouchDB does not support HTTP 1.1 ? - Andrey
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090083#comment-13090083
]
James Cohen commented on COUCHDB-1258:
--
Overnight I ran a dump/load of the database with the Python tools couchdb-dump
and couchdb-load, I now have
"doc_count":593274,"doc_del_count":0,"update_seq":593274
The views are generated with minimal memory usage (far better than in either
case before) and the problem doesn't seem to be present.
emitting null instead of the doc results in high memory usage by beam.smp and
ultimately the same OOM error
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1153) Database and view index compaction daemon
[ https://issues.apache.org/jira/browse/COUCHDB-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090078#comment-13090078 ] Jan Lehnardt commented on COUCHDB-1153: --- Paul, this came across wrong, sorry for the added confusion and feelings hurt. My post was aimed at "let's get back to work", not disregarding your efforts. Just to close the loop, I was going through this ticket and dev@ looking for "open issues" and then categorised them into "this code needs work" and "this code needs work and thus should not be in trunk at this point" (with the obvious goal of removing all of the latter ones, so we can retroactively have the patch in trunk okayed). I used the catch-all "bikedshed" for the former category and hence totally disregarded your comments. I was meaning to say that these aren't blockers (in my opinion) that would warrant a revert of the patch and I didn't mean to imply that this is some opinion you pulled out of thin air. Sorry, again. I hope we can put this to rest now. > Database and view index compaction daemon > - > > Key: COUCHDB-1153 > URL: https://issues.apache.org/jira/browse/COUCHDB-1153 > Project: CouchDB > Issue Type: New Feature > Environment: trunk >Reporter: Filipe Manana >Assignee: Filipe Manana >Priority: Minor > Labels: compaction > > I've recently written an Erlang process to automatically compact databases > and they're views based on some configurable parameters. These parameters can > be global or per database and are: minimum database fragmentation, minimum > view fragmentation, allowed period and "strict_window" (whether an ongoing > compaction should be canceled if it doesn't finish within the allowed > period). These fragmentation values are based on the recently added > "data_size" parameter to the database and view group information URIs > (COUCHDB-1132). > I've documented the .ini configuration, as a comment in default.ini, which I > paste here: > [compaction_daemon] > ; The delay, in seconds, between each check for which database and view > indexes > ; need to be compacted. > check_interval = 60 > ; If a database or view index file is smaller then this value (in bytes), > ; compaction will not happen. Very small files always have a very high > ; fragmentation therefore it's not worth to compact them. > min_file_size = 131072 > [compactions] > ; List of compaction rules for the compaction daemon. > ; The daemon compacts databases and they're respective view groups when all > the > ; condition parameters are satisfied. Configuration can be per database or > ; global, and it has the following format: > ; > ; database_name = parameter=value [, parameter=value]* > ; _default = parameter=value [, parameter=value]* > ; > ; Possible parameters: > ; > ; * db_fragmentation - If the ratio (as an integer percentage), of the amount > ; of old data (and its supporting metadata) over the > database > ; file size is equal to or greater then this value, this > ; database compaction condition is satisfied. > ; This value is computed as: > ; > ; (file_size - data_size) / file_size * 100 > ; > ; The data_size and file_size values can be obtained when > ; querying a database's information URI (GET /dbname/). > ; > ; * view_fragmentation - If the ratio (as an integer percentage), of the > amount > ;of old data (and its supporting metadata) over the > view > ;index (view group) file size is equal to or greater > then > ;this value, then this view index compaction > condition is > ;satisfied. This value is computed as: > ; > ;(file_size - data_size) / file_size * 100 > ; > ;The data_size and file_size values can be obtained > when > ;querying a view group's information URI > ;(GET /dbname/_design/groupname/_info). > ; > ; * period - The period for which a database (and its view groups) compaction > ;is allowed. This value must obey the following format: > ; > ;HH:MM - HH:MM (HH in [0..23], MM in [0..59]) > ; > ; * strict_window - If a compaction is still running after the end of the > allowed > ; period, it will be canceled if this parameter is set to > "yes". > ; It defaults to "no" and it's meaningful only if the > *period* > ; parameter is also specified. > ; > ; * parallel_view_compaction - If set to "yes", the database and its views are > ; compacted in parallel. This
[jira] [Updated] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds updated COUCHDB-1258:
---
Affects Version/s: 1.1
Surely one of these two views emits many more documents than the other and
there is something about that which is exhausting heap space. The negation in
JavaScript cannot affect the memory usage of BEAM because the view server runs
in a separate process with its own heap and the errors you see are from Erlang.
Someone will have to take a deep look into the view code and audit it for any
sloppy memory usage.
As an experiment could you try emitting null as the value rather than doc? If
the docs are very big perhaps we need to flush them in smaller batches or
something.
> eheap_alloc OOM errors when attempting to build selected views
> --
>
> Key: COUCHDB-1258
> URL: https://issues.apache.org/jira/browse/COUCHDB-1258
> Project: CouchDB
> Issue Type: Bug
>Affects Versions: 1.1
> Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
> CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
> Couch database exhibiting this behaviour:
> {"db_name":"activity_new","doc_count":593274,"doc_del_count":4743352,"update_seq":10559287,"purge_seq":0,"compact_running":false,"disk_size":3366396013,"instance_start_time":"1314097423985726","disk_format_version":5,"committed_update_seq":10559287}
>Reporter: James Cohen
>Priority: Critical
>
> We spotted OOM errors crashing our CouchDb instance when attempting to
> rebuild selected views. CouchDb was dying with the following messages (worth
> noting that the type reported varies between heap/old_heap
> eheap_alloc: Cannot allocate 478288480 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "heap").
> eheap_alloc: Cannot allocate 747325720 bytes of memory (of type "old_heap").
> eheap_alloc: Cannot allocate 597860600 bytes of memory (of type "old_heap").
> By modifying the view I was able to find a view that could consistently crash
> the server and another that ran fine. They are as follows:
> Runs out of memory v.quickly
> {
>"_id": "_design/cleanup",
>"_rev": "5-e004fbab278355e9d08763877e5a8295",
>"views": {
>"byDate": {
>"map": "function(doc) { if (! doc.action) emit([doc.date], doc); }"
>}
>}
> }
> Runs fine with minimal memory usage (returns 88128 docs in the view)
> {
>"_id": "_design/cleanup",
>"_rev": "6-3823be6b72ca2441e235addfece6900c",
>"views": {
>"byDate": {
>"map": "function(doc) { if (doc.action) emit([doc.date], doc); }"
>}
>}
> }
> The only difference between the two is the negation of the if conditional.
> memory usage was monitored with top on the machine while the view was being
> built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
> server's memory. With the view that causes problems that memory usage
> increased until the RAM/swap on the server was exhausted (as you can see from
> the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (COUCHDB-1153) Database and view index compaction daemon
On Tuesday, August 23, 2011, Damien Katz (JIRA) wrote: > >[ https://issues.apache.org/jira/browse/COUCHDB-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089512#comment-13089512] > > Damien Katz commented on COUCHDB-1153: > -- > > Robert, Benoit, your issues can still be addressed. You can submit patches that improve upon Filipes work. But telling Filipe to code the patch your way, without code is not how this community works. Filipe's work is a feature people care about, and any objections of correctness have been addressed. Switching the code to an evented model, or any other improvements is welcome from you or any other community member, but users want this feature, and Filipe should not be expected to code it up to everyone else expectations before any check-in can occur. Improvement can, and should, happen continuously. > Code correctness != it works. Anyway If we make it a rule I'm OK with that.
