[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13177733#comment-13177733 ] Henrik Hofmeister commented on COUCHDB-1367: Although its hard not to agree with Robert on c-l - i still can't really see a use-case for the update_seq value - specifically why one would ever want to know that revs_limit etc. has been updated - without being able to know what has been updated. Just Something has been done - 5 times ... ? I'd say for the sake of expected output - c-l aside - this is still a bug / hidden feature - and should at the very least be documented ? When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 27, 2011 at 22:37, Jason Smith j...@iriscouch.com wrote: On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Your idea improves consistency and orthogonality. It also solves the problem of how to enumerate _local docs. (AFAIK there is no way to list them all, not via _all_docs, or _changes, or a view). But it doesn't solve the larger problem: How to follow a _changes feed and know when you have caught up. Both Bob N. and I independently did the following for our projects: 1. GET /db and wrongly assume update_seq will appear in the changes feed 2. GET /db/_changes?feed=continuous 3. Break when a change has .seq = update_seq Suppose you have step 0: Update _security or _revs_limit. The loop will never break. You propose (WLOG) _changes?comprehensive=true which guarantees a change equal or greater than update_seq. That's cool, but IMO app developers now have to add code to ignore irrelevant changes like those containing replication checkpoints. All great points. I propose (WLOG) update_sikh in the db header which is the seq id of the latest *document* update. App developers modify their step 1 to use update_sikh instead of update_seq. Is that an accurate synopsis? Yes. If we decide to go this route I would rather see _revs_limit and _security stop bumping update_seq because I find it confusing that update_seq is then not directly related to the by_seq tree. Granted, app developers don't need to know anything about the by_seq tree, but I don't see a compelling reason to keep the old definition of update_seq. The one exception raised so far is your observation that it may be useful for doing backups. However, if that's the only real exception, perhaps we should surface some other thing more directly analogous to the UNIX ctime. Is it really true that shards need the same _revs_limit as the simulated whole? Maybe they really want _revs_limit / number_of_shards? Is it really necessary that _security be identical in each shard? Actually, yes it is, because validate_doc_update uses it. But still... How are you computing doc_count in the /db response? You have to sum doc_count from each shard. But every shard needs a copy of every design doc for validation. So you have to subtract those back out? My broader point is, sharding applications already do lots of magic. I'm not sure if replicating _security and _local docs buys you much. Good observations as well. -Randall
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Thu, Dec 29, 2011 at 1:44 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 22:37, Jason Smith j...@iriscouch.com wrote: On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Your idea improves consistency and orthogonality. It also solves the problem of how to enumerate _local docs. (AFAIK there is no way to list them all, not via _all_docs, or _changes, or a view). But it doesn't solve the larger problem: How to follow a _changes feed and know when you have caught up. Both Bob N. and I independently did the following for our projects: 1. GET /db and wrongly assume update_seq will appear in the changes feed 2. GET /db/_changes?feed=continuous 3. Break when a change has .seq = update_seq Suppose you have step 0: Update _security or _revs_limit. The loop will never break. You propose (WLOG) _changes?comprehensive=true which guarantees a change equal or greater than update_seq. That's cool, but IMO app developers now have to add code to ignore irrelevant changes like those containing replication checkpoints. All great points. I propose (WLOG) update_sikh in the db header which is the seq id of the latest *document* update. App developers modify their step 1 to use update_sikh instead of update_seq. Is that an accurate synopsis? Yes. If we decide to go this route I would rather see _revs_limit and _security stop bumping update_seq because I find it confusing that update_seq is then not directly related to the by_seq tree. update_seq is not directly related to the by_seq tree. That is a very powerful observation. My idea has less orthogonality and conceptual simplicity. The only mystery is why update_seq did this in the first place. Did Damien have a reason, from hard-won experience? The one exception raised so far is your observation that it may be useful for doing backups. However, if that's the only real exception, perhaps we should surface some other thing more directly analogous to the UNIX crime. Meh, the backups argument is okay, but weak. If you depend on it, you'll miss _local docs (replication checkpoints, so that's no small error). You still have to back up .ini files, and perhaps logs and view files. So my suggestion for Unix crime is just that: OS timestamps. -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Dec 28, 2011, at 4:21 PM, Jason Smith wrote: On Thu, Dec 29, 2011 at 1:44 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 22:37, Jason Smith j...@iriscouch.com wrote: On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Your idea improves consistency and orthogonality. It also solves the problem of how to enumerate _local docs. (AFAIK there is no way to list them all, not via _all_docs, or _changes, or a view). But it doesn't solve the larger problem: How to follow a _changes feed and know when you have caught up. Both Bob N. and I independently did the following for our projects: 1. GET /db and wrongly assume update_seq will appear in the changes feed 2. GET /db/_changes?feed=continuous 3. Break when a change has .seq = update_seq Suppose you have step 0: Update _security or _revs_limit. The loop will never break. You propose (WLOG) _changes?comprehensive=true which guarantees a change equal or greater than update_seq. That's cool, but IMO app developers now have to add code to ignore irrelevant changes like those containing replication checkpoints. All great points. I propose (WLOG) update_sikh in the db header which is the seq id of the latest *document* update. App developers modify their step 1 to use update_sikh instead of update_seq. Is that an accurate synopsis? Yes. If we decide to go this route I would rather see _revs_limit and _security stop bumping update_seq because I find it confusing that update_seq is then not directly related to the by_seq tree. update_seq is not directly related to the by_seq tree. That is a very powerful observation. My idea has less orthogonality and conceptual simplicity. The only mystery is why update_seq did this in the first place. Did Damien have a reason, from hard-won experience? I wanted to give an easy way to monitor other changes to the database header, for things like automated backup and admin tool UIs, though I don't know if anything uses it. It was easiest to do it this way, and I ddn't think about the problems it might cause with users expecting it to correspond with the by_seq index. With all the confusion it's caused, I'd be fine with taking it out. Or maybe put in a db version_seq header, that's like a changes feed but for all types of updates to the db. The one exception raised so far is your observation that it may be useful for doing backups. However, if that's the only real exception, perhaps we should surface some other thing more directly analogous to the UNIX crime. Meh, the backups argument is okay, but weak. If you depend on it, you'll miss _local docs (replication checkpoints, so that's no small error). You still have to back up .ini files, and perhaps logs and view files. So my suggestion for Unix crime is just that: OS timestamps. -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Wed, Dec 28, 2011 at 7:02 PM, Damien Katz dam...@apache.org wrote: On Dec 28, 2011, at 4:21 PM, Jason Smith wrote: On Thu, Dec 29, 2011 at 1:44 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 22:37, Jason Smith j...@iriscouch.com wrote: On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Your idea improves consistency and orthogonality. It also solves the problem of how to enumerate _local docs. (AFAIK there is no way to list them all, not via _all_docs, or _changes, or a view). But it doesn't solve the larger problem: How to follow a _changes feed and know when you have caught up. Both Bob N. and I independently did the following for our projects: 1. GET /db and wrongly assume update_seq will appear in the changes feed 2. GET /db/_changes?feed=continuous 3. Break when a change has .seq = update_seq Suppose you have step 0: Update _security or _revs_limit. The loop will never break. You propose (WLOG) _changes?comprehensive=true which guarantees a change equal or greater than update_seq. That's cool, but IMO app developers now have to add code to ignore irrelevant changes like those containing replication checkpoints. All great points. I propose (WLOG) update_sikh in the db header which is the seq id of the latest *document* update. App developers modify their step 1 to use update_sikh instead of update_seq. Is that an accurate synopsis? Yes. If we decide to go this route I would rather see _revs_limit and _security stop bumping update_seq because I find it confusing that update_seq is then not directly related to the by_seq tree. update_seq is not directly related to the by_seq tree. That is a very powerful observation. My idea has less orthogonality and conceptual simplicity. The only mystery is why update_seq did this in the first place. Did Damien have a reason, from hard-won experience? I wanted to give an easy way to monitor other changes to the database header, for things like automated backup and admin tool UIs, though I don't know if anything uses it. It was easiest to do it this way, and I ddn't think about the problems it might cause with users expecting it to correspond with the by_seq index. With all the confusion it's caused, I'd be fine with taking it out. Or maybe put in a db version_seq header, that's like a changes feed but for all types of updates to the db. Having a _changes and _events feed I think makes good sense. It could get us away from the db update notifier stuff as well if we consolidated things in there. Though as a note we need to remember to fix the triggering mechanism for _view updates because of _purge. I think the simplest way would be to just check both the _update and _purge seq's instead of just relying on _purge to have changed the _update seq. Luckily we store both of these thigns already so it should just be a matter of updating the logic in the guards when grabbing a view group. The one exception raised so far is your observation that it may be useful for doing backups. However, if that's the only real exception, perhaps we should surface some other thing more directly analogous to the UNIX crime. Meh, the backups argument is okay, but weak. If you depend on it, you'll miss _local docs (replication checkpoints, so that's no small error). You still have to back up .ini files, and perhaps logs and view files. So my suggestion for Unix crime is just that: OS timestamps. -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Yes it does. There is mostly consistent relationship between update sequence (seq, update_seq, last_seq, committed_seq) and the by_seq index. It seems entirely too confusing that there are things which affect update_seq but do not appear in the by_seq btree. That is just plain wrong, else a massive confusion of vocabulary. I think it is confused definitions. If by_seq is defined as the sequence ids of regular documents then it is implemented correctly. Bear with me for I believe ther is a related discussion about replicability for _security, _local docs, etc. It's clear that there are clustering and operational motivations for making this information replicable, thus making them proper documents with a place in the by_seq index, in the _changes feed, and affecting update_seq. It is common to track changes to data vs. changes to metadata independently. Changing Unix file permissions updates ctime but not mtime. Changing Unix files updates both ctime and mtime. In CouchDB, update_seq plays the role of ctime (data or metadata updates), and nobody's been cast for mtime (metadata-only updates). Either these things have a proper place in the sequential history of a database or they do not. That there are things which affect update_seq but do not appear in the by_seq index and _changes feed feels like a mistake. The first sentence is, well, a tautology actually, but it asks the right question and the answer is they DO NOT belong. _changes shows data, not metadata. By definition, _changes is anything worth replicating. But I hope my filesystem example above shows why it is okay to increment update_seq but not change by_seq. The bug with update_seq is not that it it is too eager (increments for _security, _revs_limit), but it is not eager enough (it should bump for _local too). 2. As a frequent consumer of _changes, I would prefer *not* to see _local documents, nor _security or other updates in there. They are metadata, not data. Maybe I misunderstood, but nobody wants to *replicate* _security objects or _local docs; they just want MVCC semantics (Adam on _security, IIRC) and a simplified API (me, on making all metadata a _local doc, and making _local docs full MVCC). Placing additional metadata in the db header feels like rubbing salt in this wound. On the contrary, IMHO, we want 1. A new value: the sequence id of the most recent document update (pretty sure) 2. Available to the client alongside existing values like doc_count doc_del_count (somewhat sure) Right now only replicable documents surface in the _changes feed and are added to the by_seq btree but some other things affect the update_seq. I've just gone and checked, as described in my previous email, that none of these appear to require a change to update_seq for any technical reason, though Jason properly points out that it is perhaps useful for operational tasks such as knowing when to back up a .couch file. Here is where get into migrating to more _local docs. I am actually not sure if that's good for this discussion. But anyway, my basic feeling is * All metadata that clients can change is _local docs, with MVCC, *not in* the by_seq tree * update_seq counts changes to data or metadata * update_sikh (WLOG) counts changes to documents only (changes to the by_seq tree) Doable? It bears mentioning that I haven't any idea what I am talking about. Thoughts, concerns, emotions and relevant, famous quotations encouraged. WELCOME TO THE PARTY, PAL! -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
What's interesting is that modulo one edge case, last_seq in the changes feed
and update_seq in the db_info record are exactly as defined on the WIKI.
update_seq:
Current number of updates to the database (int)
last_seq:
last_seq is the sequence number of the last update returned. (Currently it will
always be the same as the seq of the last item in results.)
this holds true also when there are no changes to documents, the value of
last_seq is zero. The one edge case (which is a bit odd) is seen when you
retrieve last_seq using ?descending=truelimit=1. If there are no changes the
value will still be zero unless you call _set_revs_limit first in which case
the value will be one. The value will still be zero if the normal _changes is
called with no args. What makes it odd is that calling _changes?descending...
after a call to _set_revs_limit does not impact the value of last_seq. This is
a bug.
So yes it's a bit weird but it does pretty much agree with the documentation.
The quote I'm looking for is the one about angels on the head of a pin.
I guess it needs more thought. In general I don't like metadata because I think
it creates more things that need to be handled differently, adding complexity
for the sake of something that doesn't exist (metadata).
Do you have any more swatches in magenta?
On Dec 27, 2011, at 12:04 AM, Randall Leeds wrote:
On Mon, Dec 26, 2011 at 08:49, Jason Smith j...@iriscouch.com wrote:
Hi, Bob. Thanks for your feedback.
On Mon, Dec 26, 2011 at 12:24 PM, Robert Dionne
dio...@dionne-associates.com wrote:
Jason,
After looking into this a bit I do not think it's a bug, at most poor
documentation. update_seq != last_seq
Nobody knows what update_seq means. Even a CouchDB committer got it wrong.
Fine. It is poor documentation.
Adding last_seq into db_info is not helpful because last_seq also does
not mean what we think it means. My last email demonstrates that
last_seq is in fact incoherent.
snip
On Mon, Dec 26, 2011 at 23:03, Benoit Chesneau bchesn...@gmail.com wrote:
Mmm right that confusing (maybe except if you consider update_seq as a
way to know the numbers of updates in the databases but in this case
the wording is confiusing) . Imo changes seq commited_seq should be
quites the same. At least a changes seq should only happen when there
is a doc update ie each time and only if a revision is created. Does
that make sense?
- benoiît
Yes it does. There is mostly consistent relationship between update
sequence (seq, update_seq, last_seq, committed_seq) and the by_seq
index. It seems entirely too confusing that there are things which
affect update_seq but do not appear in the by_seq btree. That is just
plain wrong, else a massive confusion of vocabulary. Benoit, I believe
you are right to suggest that none of these sequences-related things
should change unless a revision is created.
Bear with me for I believe ther is a related discussion about
replicability for _security, _local docs, etc. It's clear that there
are clustering and operational motivations for making this information
replicable, thus making them proper documents with a place in the
by_seq index, in the _changes feed, and affecting update_seq. Either
these things have a proper place in the sequential history of a
database or they do not. That there are things which affect update_seq
but do not appear in the by_seq index and _changes feed feels like a
mistake. Placing additional metadata in the db header feels like
rubbing salt in this wound.
Right now only replicable documents surface in the _changes feed and
are added to the by_seq btree but some other things affect the
update_seq. I've just gone and checked, as described in my previous
email, that none of these appear to require a change to update_seq for
any technical reason, though Jason properly points out that it is
perhaps useful for operational tasks such as knowing when to back up a
.couch file.
I see two reasonable ways forward.
1) Stop incrementing update_seq for anything but replicable document changes
2) Make things which already affect update_seq but do not appear in
_changes appear there, likely by turning them into proper MVCC
documents.
Regarding option 1:
This is easy. I already outlined how to do this. It requires removing
about 3 characters from our codebase. However, it spits at Jason's
operations concerns, which I think are quite valid, and misses an
opportunity for great improvement.
Regarding option 2:
There is a cluster-aware use case, an operations use case, and, I
think, a purity argument here. As for how to accomplish this feat
without terrible API breakage, we get a lot of help from our URL
structure. We have reserved paths which cannot conflict with documents
so it does not create ambiguity if '{seq:20,id:_security, ...}'
appears in a changes feed. However, I think _security is a bad name
for this document because it requires
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Either these things have a proper place in the sequential history of a database or they do not. That there are things which affect update_seq but do not appear in the by_seq index and _changes feed feels like a mistake. The first sentence is, well, a tautology actually, but it asks the right question and the answer is they DO NOT belong. _changes shows data, not metadata. By definition, _changes is anything worth replicating. That strikes me as incorrect. The _changes feed is purely metadata unless ?include_docs=true is specified. But I hope my filesystem example above shows why it is okay to increment update_seq but not change by_seq. You show a nice precedent for separating metadata and data, but CouchDB has a decent precedent of avoiding this same thing. For example, _id and _rev are in the returned document body rather than part of the HTTP request (it could have been just URL and entity tag headers only for this). The bug with update_seq is not that it it is too eager (increments for _security, _revs_limit), but it is not eager enough (it should bump for _local too). I agree, but for different reasons. I think _local docs may have a place in by_seq even if the default _changes request still only shows the default, replicable documents. 2. As a frequent consumer of _changes, I would prefer *not* to see _local documents, nor _security or other updates in there. They are metadata, not data. Maybe I misunderstood, but nobody wants to *replicate* _security objects or _local docs; they just want MVCC semantics (Adam on _security, IIRC) and a simplified API (me, on making all metadata a _local doc, and making _local docs full MVCC). I think you misunderstand, maybe. In the case of BigCouch, MVCC is all that's needed because the replication does not go over HTTP. I see no reason to require that special care be taken to copy these objects when a flag on the _changes feed might cause them to be transferred very naturally. In particular, I would use this feature in a hypothetical Lounge 3.0. It also means that with admin privileges we could do full backup replications. -R
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Wed, Dec 28, 2011 at 9:38 AM, Randall Leeds randall.le...@gmail.com wrote: On Tue, Dec 27, 2011 at 05:22, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 5:04 AM, Randall Leeds randall.le...@gmail.com wrote: Either these things have a proper place in the sequential history of a database or they do not. That there are things which affect update_seq but do not appear in the by_seq index and _changes feed feels like a mistake. The first sentence is, well, a tautology actually, but it asks the right question and the answer is they DO NOT belong. _changes shows data, not metadata. By definition, _changes is anything worth replicating. That strikes me as incorrect. The _changes feed is purely metadata unless ?include_docs=true is specified. Yes, data and metadata are problematic words. I'll stop using them. Do you agree that _changes is, by definition, anything worth replicating? But I hope my filesystem example above shows why it is okay to increment update_seq but not change by_seq. You show a nice precedent for separating metadata and data, but CouchDB has a decent precedent of avoiding this same thing. For example, _id and _rev are in the returned document body rather than part of the HTTP request (it could have been just URL and entity tag headers only for this). Yeah that's a good point. The bug with update_seq is not that it it is too eager (increments for _security, _revs_limit), but it is not eager enough (it should bump for _local too). I agree, but for different reasons. I think _local docs may have a place in by_seq even if the default _changes request still only shows the default, replicable documents. That's an interesting idea. IMO, _security, _revs_limit, apply to a specific database and URL, and consequently must never replicate. _local docs are those which don't replicate. If _local would replicate, I'd worry about spurious checkpoints spreading to where they don't belong; and unchecked _security replication is even worse. Your idea improves consistency and orthogonality. It also solves the problem of how to enumerate _local docs. (AFAIK there is no way to list them all, not via _all_docs, or _changes, or a view). But it doesn't solve the larger problem: How to follow a _changes feed and know when you have caught up. Both Bob N. and I independently did the following for our projects: 1. GET /db and wrongly assume update_seq will appear in the changes feed 2. GET /db/_changes?feed=continuous 3. Break when a change has .seq = update_seq Suppose you have step 0: Update _security or _revs_limit. The loop will never break. You propose (WLOG) _changes?comprehensive=true which guarantees a change equal or greater than update_seq. That's cool, but IMO app developers now have to add code to ignore irrelevant changes like those containing replication checkpoints. I propose (WLOG) update_sikh in the db header which is the seq id of the latest *document* update. App developers modify their step 1 to use update_sikh instead of update_seq. Is that an accurate synopsis? 2. As a frequent consumer of _changes, I would prefer *not* to see _local documents, nor _security or other updates in there. They are metadata, not data. Maybe I misunderstood, but nobody wants to *replicate* _security objects or _local docs; they just want MVCC semantics (Adam on _security, IIRC) and a simplified API (me, on making all metadata a _local doc, and making _local docs full MVCC). I think you misunderstand, maybe. In the case of BigCouch, MVCC is all that's needed because the replication does not go over HTTP. I see no reason to require that special care be taken to copy these objects when a flag on the _changes feed might cause them to be transferred very naturally. In particular, I would use this feature in a hypothetical Lounge 3.0. It also means that with admin privileges we could do full backup replications. If couch could do this, then cool. But consider that both examples are the same special-case: sharding and simulating a normal database API when there are actually multiple parts. That sounds like an application concern. Is it really true that shards need the same _revs_limit as the simulated whole? Maybe they really want _revs_limit / number_of_shards? Is it really necessary that _security be identical in each shard? Actually, yes it is, because validate_doc_update uses it. But still... How are you computing doc_count in the /db response? You have to sum doc_count from each shard. But every shard needs a copy of every design doc for validation. So you have to subtract those back out? My broader point is, sharding applications already do lots of magic. I'm not sure if replicating _security and _local docs buys you much. But you've definitely persuaded me that your idea works. It is the second-best proposal I've seen in this thread :) -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
Hi, Randall. Thanks for inviting me to argue a bit more. I hope you'll
be persuaded that, if -1367 is not a bug, at least there is *some*
bug.
tl;dr summary:
This is a real bug--a paper cut with a workaround, but still a real bug.
1. Apps want a changes feed since 0, but they want to know when
they've caught up (defined below)
2. These apps (and robust apps generally) probably start out by
pinging the /db anyway. Bob N. and I independently did so.
3. update_seq looks deceptively like the sequence id of the latest
change, and people assume so. They define caught up as receiving a
change at or above this value. They expect to catch up in finite
time, and even if the db receives no subsequent updates.
4. In fact, CouchDB does not disclose the sequence id of the latest
change in the /db response. To know that value:
4a. If you want to process every change anyway, just get _changes
and use last_seq
4b. If you just want the last sequence id, query
_changes?descending=truelimit=1
4b(1). If the response has a change, use its last_seq value
4b(2). If the response has no changes, ignore the last_seq value
(it is really the update_seq) and use 0
Step 3 is the major paper cut. That step 4 exists and is complicated
is the minor paper cut.
On Mon, Dec 26, 2011 at 5:36 AM, Randall Leeds (Commented) (JIRA)
j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175892#comment-13175892
]
Randall Leeds commented on COUCHDB-1367:
Wait a second. Robert, you are not fixing a bug in C-L, you are working
around a deficiency in CouchDB.
Can't both be true?
Only in the trivial sense. This ticket reveals that app
developers--Henrik and me, but also a committer--misunderstand
update_seq, thinking it is last_seq. last_seq is not easy to learn.
Nope. You can not ever know. You always know the latest sequence number at
some arbitrarily recent point in time.
Sorry, I cut corners and was not clear. Of course, nobody ever really
knows anything except events in the very recent past. But I mean in
the context of a _changes query one-two punch: get the last_seq, then
begin a continuous feed since that value.
The bug is that users cannot readily know the id of the most recent
change. In fact, the id of the most recent change has no explicit
label or name in the CouchDB interface. Neither update_seq nor
last_seq mean exactly that.
What if I want to see the most recent five changes? What if there are a
hundred million documents? What if 99% of the time, update_seq equals
last_seq and so developers assume it means something it doesn't?
In order:
* /_changes?descending=truelimit=5
I stand corrected. I had forgotten about a descending changes query.
That resolves the hundred-million-docs problem. (My erroneous point
was, 100M docs makes it too expensive to learn last_seq.)
But that response looks bizarre.
GET /db/_changes?descending=true\limit=5
{results:[
{seq:22,id:after_3,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:21,id:after_2,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:20,id:after_1,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:19,id:conc,changes:[{rev:2-584a4a504a97009241d2587fee8b5eb8}]},
{seq:17,id:preload_create,changes:[{rev:1-28bf6cd8af83c40c6e3fb82b608ce98f}]}
],
last_seq:17}
last_seq is the *least recent* change. If you query with limit=1 then
they will be equal, and that is nice. *Except* if there were no
changes yet.
$ curl -X PUT localhost:5984/x
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl localhost:5984/x/_changes
{results:[
],
last_seq:0}
$ curl localhost:5984/x/_changes?descending=true
{results:[
],
last_seq:3}
Weird.
* Add additional information to the changes feed, perhaps with a query
parameter (almost the reverse of include docs)
* Stop incrementing the update sequence on certain kinds of non-document
changes
* Add more information to the db information response
A commonly-needed and valuable piece of data like this seems most
appropriate cached in the db header and served in the db information.
--
Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
Jason,
After looking into this a bit I do not think it's a bug, at most poor
documentation. update_seq != last_seq Most of the time it does but as we know
now sometimes it doesn't. It's a different thing. Im not sure where else in
the code we depend on update_seq reflecting all the changes to the database,
perhaps as Randall suggests we might be able to *not* bump it in those other
calls.
Another way to handle this is hang on to the last_seq when a changes call is
made and use that as a since parameter in the next call. This to me seems like
what's needed in this use case anyway.
In any event it's likely easy to add last_seq to the db_info record, and I'm
more than happy to do that, we should open a new ticket for that.
Cheers,
Bob
On Dec 26, 2011, at 4:10 AM, Jason Smith wrote:
Hi, Randall. Thanks for inviting me to argue a bit more. I hope you'll
be persuaded that, if -1367 is not a bug, at least there is *some*
bug.
tl;dr summary:
This is a real bug--a paper cut with a workaround, but still a real bug.
1. Apps want a changes feed since 0, but they want to know when
they've caught up (defined below)
2. These apps (and robust apps generally) probably start out by
pinging the /db anyway. Bob N. and I independently did so.
3. update_seq looks deceptively like the sequence id of the latest
change, and people assume so. They define caught up as receiving a
change at or above this value. They expect to catch up in finite
time, and even if the db receives no subsequent updates.
4. In fact, CouchDB does not disclose the sequence id of the latest
change in the /db response. To know that value:
4a. If you want to process every change anyway, just get _changes
and use last_seq
4b. If you just want the last sequence id, query
_changes?descending=truelimit=1
4b(1). If the response has a change, use its last_seq value
4b(2). If the response has no changes, ignore the last_seq value
(it is really the update_seq) and use 0
Step 3 is the major paper cut. That step 4 exists and is complicated
is the minor paper cut.
On Mon, Dec 26, 2011 at 5:36 AM, Randall Leeds (Commented) (JIRA)
j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175892#comment-13175892
]
Randall Leeds commented on COUCHDB-1367:
Wait a second. Robert, you are not fixing a bug in C-L, you are working
around a deficiency in CouchDB.
Can't both be true?
Only in the trivial sense. This ticket reveals that app
developers--Henrik and me, but also a committer--misunderstand
update_seq, thinking it is last_seq. last_seq is not easy to learn.
Nope. You can not ever know. You always know the latest sequence number at
some arbitrarily recent point in time.
Sorry, I cut corners and was not clear. Of course, nobody ever really
knows anything except events in the very recent past. But I mean in
the context of a _changes query one-two punch: get the last_seq, then
begin a continuous feed since that value.
The bug is that users cannot readily know the id of the most recent
change. In fact, the id of the most recent change has no explicit
label or name in the CouchDB interface. Neither update_seq nor
last_seq mean exactly that.
What if I want to see the most recent five changes? What if there are a
hundred million documents? What if 99% of the time, update_seq equals
last_seq and so developers assume it means something it doesn't?
In order:
* /_changes?descending=truelimit=5
I stand corrected. I had forgotten about a descending changes query.
That resolves the hundred-million-docs problem. (My erroneous point
was, 100M docs makes it too expensive to learn last_seq.)
But that response looks bizarre.
GET /db/_changes?descending=true\limit=5
{results:[
{seq:22,id:after_3,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:21,id:after_2,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:20,id:after_1,changes:[{rev:1-0785e9eb543380151003dc452c3a001a}]},
{seq:19,id:conc,changes:[{rev:2-584a4a504a97009241d2587fee8b5eb8}]},
{seq:17,id:preload_create,changes:[{rev:1-28bf6cd8af83c40c6e3fb82b608ce98f}]}
],
last_seq:17}
last_seq is the *least recent* change. If you query with limit=1 then
they will be equal, and that is nice. *Except* if there were no
changes yet.
$ curl -X PUT localhost:5984/x
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl -X PUT localhost:5984/x/_revs_limit -d $RANDOM
{ok:true}
$ curl localhost:5984/x/_changes
{results:[
],
last_seq:0}
$ curl localhost:5984/x/_changes?descending=true
{results:[
],
last_seq:3}
Weird.
* Add additional information to the changes feed,
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
Hi, Bob. Thanks for your feedback. On Mon, Dec 26, 2011 at 12:24 PM, Robert Dionne dio...@dionne-associates.com wrote: Jason, After looking into this a bit I do not think it's a bug, at most poor documentation. update_seq != last_seq Nobody knows what update_seq means. Even a CouchDB committer got it wrong. Fine. It is poor documentation. Adding last_seq into db_info is not helpful because last_seq also does not mean what we think it means. My last email demonstrates that last_seq is in fact incoherent. What would be marvelous in the db_info is this: The sequence id of the most recent document update, or zero if there has not been one. If you add it, that would be great but you might first consider what its relationship is to last_seq, why the values differ sometimes, and whether that is good for users. Thanks again! -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Mon, Dec 26, 2011 at 08:49, Jason Smith j...@iriscouch.com wrote: Hi, Bob. Thanks for your feedback. On Mon, Dec 26, 2011 at 12:24 PM, Robert Dionne dio...@dionne-associates.com wrote: Jason, After looking into this a bit I do not think it's a bug, at most poor documentation. update_seq != last_seq Nobody knows what update_seq means. Even a CouchDB committer got it wrong. Fine. It is poor documentation. Adding last_seq into db_info is not helpful because last_seq also does not mean what we think it means. My last email demonstrates that last_seq is in fact incoherent. Awesome. I'm glad you testing descending. Sounds like last_seq is a poor name, because it applies to the particular changes request. So then we have this other thing floating around the sequence number of the last replicable document change. Interestingly, updates to _local/id documents don't affect update_seq. Looking into the code, I see all the places where it's bumped artificially. It's quite obvious, actually. Search for update_seq+1 in couch_db_updater.erl. 1) On setting _revs_limit 2) On setting _security 3) On some call to increment it (who knows why) that has an HTTP POST handler in couch_httpd_misc_handlers that is not exposed by etc/couchdb/default.ini. I don't see any reason why (1) and (2) need to be bumping this number. (3) has been there for a long time but doesn't seem to be part of the default public API. It appears to have been introduced by Damien in May of 2008 (333d18cf) with the commit message: Experimental functionality to increment database update seq, might go away, use at own risk. I propose we just get rid of all these and then update_seq becomes what everyone expects it to be. -Randall
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 27, 2011 at 9:02 AM, Randall Leeds randall.le...@gmail.com wrote: Awesome. I'm glad you testing descending. Sounds like last_seq is a poor name, because it applies to the particular changes request. So then we have this other thing floating around the sequence number of the last replicable document change. Interestingly, updates to _local/id documents don't affect update_seq. Looking into the code, I see all the places where it's bumped artificially. It's quite obvious, actually. Search for update_seq+1 in couch_db_updater.erl. 1) On setting _revs_limit 2) On setting _security 3) On some call to increment it (who knows why) that has an HTTP POST handler in couch_httpd_misc_handlers that is not exposed by etc/couchdb/default.ini. I don't see any reason why (1) and (2) need to be bumping this number. (3) has been there for a long time but doesn't seem to be part of the default public API. It appears to have been introduced by Damien in May of 2008 (333d18cf) with the commit message: Experimental functionality to increment database update seq, might go away, use at own risk. I propose we just get rid of all these and then update_seq becomes what everyone expects it to be. It sounds like update_seq an infallible indicator of when a backup is necessary--better than the filesystem timestamp for the .couch file. Just compare its value to the one from your previous backup. You'll be sure to catch not only document updates, but also per-db configuration settings like _security and _revs_limit. I'm not sure if that's 100% correct but if it is, it's useful, or at least not useless. -- Iris Couch
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 27, 2011 at 3:02 AM, Randall Leeds randall.le...@gmail.com wrote: On Mon, Dec 26, 2011 at 08:49, Jason Smith j...@iriscouch.com wrote: Hi, Bob. Thanks for your feedback. On Mon, Dec 26, 2011 at 12:24 PM, Robert Dionne dio...@dionne-associates.com wrote: Jason, After looking into this a bit I do not think it's a bug, at most poor documentation. update_seq != last_seq Nobody knows what update_seq means. Even a CouchDB committer got it wrong. Fine. It is poor documentation. Adding last_seq into db_info is not helpful because last_seq also does not mean what we think it means. My last email demonstrates that last_seq is in fact incoherent. Awesome. I'm glad you testing descending. Sounds like last_seq is a poor name, because it applies to the particular changes request. So then we have this other thing floating around the sequence number of the last replicable document change. Interestingly, updates to _local/id documents don't affect update_seq. Looking into the code, I see all the places where it's bumped artificially. It's quite obvious, actually. Search for update_seq+1 in couch_db_updater.erl. 1) On setting _revs_limit 2) On setting _security 3) On some call to increment it (who knows why) that has an HTTP POST handler in couch_httpd_misc_handlers that is not exposed by etc/couchdb/default.ini. I don't see any reason why (1) and (2) need to be bumping this number. (3) has been there for a long time but doesn't seem to be part of the default public API. It appears to have been introduced by Damien in May of 2008 (333d18cf) with the commit message: Experimental functionality to increment database update seq, might go away, use at own risk. I propose we just get rid of all these and then update_seq becomes what everyone expects it to be. -Randall Mmm right that confusing (maybe except if you consider update_seq as a way to know the numbers of updates in the databases but in this case the wording is confiusing) . Imo changes seq commited_seq should be quites the same. At least a changes seq should only happen when there is a doc update ie each time and only if a revision is created. Does that make sense? - benoiît
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Mon, Dec 26, 2011 at 22:30, Jason Smith j...@iriscouch.com wrote: On Tue, Dec 27, 2011 at 9:02 AM, Randall Leeds randall.le...@gmail.com wrote: Awesome. I'm glad you testing descending. Sounds like last_seq is a poor name, because it applies to the particular changes request. So then we have this other thing floating around the sequence number of the last replicable document change. Interestingly, updates to _local/id documents don't affect update_seq. Looking into the code, I see all the places where it's bumped artificially. It's quite obvious, actually. Search for update_seq+1 in couch_db_updater.erl. 1) On setting _revs_limit 2) On setting _security 3) On some call to increment it (who knows why) that has an HTTP POST handler in couch_httpd_misc_handlers that is not exposed by etc/couchdb/default.ini. I don't see any reason why (1) and (2) need to be bumping this number. (3) has been there for a long time but doesn't seem to be part of the default public API. It appears to have been introduced by Damien in May of 2008 (333d18cf) with the commit message: Experimental functionality to increment database update seq, might go away, use at own risk. I propose we just get rid of all these and then update_seq becomes what everyone expects it to be. It sounds like update_seq an infallible indicator of when a backup is necessary--better than the filesystem timestamp for the .couch file. Just compare its value to the one from your previous backup. You'll be sure to catch not only document updates, but also per-db configuration settings like _security and _revs_limit. I'm not sure if that's 100% correct but if it is, it's useful, or at least not useless. -- Iris Couch Thanks a ton for pointing out the operations issue of backup here; I hadn't considered it. It's worth noting, as I did before, that _local docs do not affect update_seq, though that's probably not a disaster for backup scenarios unless _local is being used by something other than the replicator. -R
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Mon, Dec 26, 2011 at 08:49, Jason Smith j...@iriscouch.com wrote:
Hi, Bob. Thanks for your feedback.
On Mon, Dec 26, 2011 at 12:24 PM, Robert Dionne
dio...@dionne-associates.com wrote:
Jason,
After looking into this a bit I do not think it's a bug, at most poor
documentation. update_seq != last_seq
Nobody knows what update_seq means. Even a CouchDB committer got it wrong.
Fine. It is poor documentation.
Adding last_seq into db_info is not helpful because last_seq also does
not mean what we think it means. My last email demonstrates that
last_seq is in fact incoherent.
snip
On Mon, Dec 26, 2011 at 23:03, Benoit Chesneau bchesn...@gmail.com wrote:
Mmm right that confusing (maybe except if you consider update_seq as a
way to know the numbers of updates in the databases but in this case
the wording is confiusing) . Imo changes seq commited_seq should be
quites the same. At least a changes seq should only happen when there
is a doc update ie each time and only if a revision is created. Does
that make sense?
- benoiît
Yes it does. There is mostly consistent relationship between update
sequence (seq, update_seq, last_seq, committed_seq) and the by_seq
index. It seems entirely too confusing that there are things which
affect update_seq but do not appear in the by_seq btree. That is just
plain wrong, else a massive confusion of vocabulary. Benoit, I believe
you are right to suggest that none of these sequences-related things
should change unless a revision is created.
Bear with me for I believe ther is a related discussion about
replicability for _security, _local docs, etc. It's clear that there
are clustering and operational motivations for making this information
replicable, thus making them proper documents with a place in the
by_seq index, in the _changes feed, and affecting update_seq. Either
these things have a proper place in the sequential history of a
database or they do not. That there are things which affect update_seq
but do not appear in the by_seq index and _changes feed feels like a
mistake. Placing additional metadata in the db header feels like
rubbing salt in this wound.
Right now only replicable documents surface in the _changes feed and
are added to the by_seq btree but some other things affect the
update_seq. I've just gone and checked, as described in my previous
email, that none of these appear to require a change to update_seq for
any technical reason, though Jason properly points out that it is
perhaps useful for operational tasks such as knowing when to back up a
.couch file.
I see two reasonable ways forward.
1) Stop incrementing update_seq for anything but replicable document changes
2) Make things which already affect update_seq but do not appear in
_changes appear there, likely by turning them into proper MVCC
documents.
Regarding option 1:
This is easy. I already outlined how to do this. It requires removing
about 3 characters from our codebase. However, it spits at Jason's
operations concerns, which I think are quite valid, and misses an
opportunity for great improvement.
Regarding option 2:
There is a cluster-aware use case, an operations use case, and, I
think, a purity argument here. As for how to accomplish this feat
without terrible API breakage, we get a lot of help from our URL
structure. We have reserved paths which cannot conflict with documents
so it does not create ambiguity if '{seq:20,id:_security, ...}'
appears in a changes feed. However, I think _security is a bad name
for this document because it requires that /_security API
compatibility is broken.
One solution I like right now is to add a _meta (without loss of
generality -- insert your own preferred name) document, with the
normal MVCC document API, referenced by the by_seq index and appearing
in the _changes feed, which contains both _revs_limit and _security
while preserving the legacy, cloberring, MVCC-oblivious APIs. Voila!
No breaking changes. Keep a pointer latest revision of this document
in the database header for fast access (and perhaps cache it in
memory).
It would probably be acceptable to keep these out of a vanilla changes
request (after all, they require db admin credentials to modify, and
in the case of _security to view). Opening the door to additional
flags for _changes also allows us to provide a natural extension of
this idea to replicable _local docs for the clustering use case.
Thoughts, concerns, emotions and relevant, famous quotations encouraged.
-Randall
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175813#comment-13175813 ] Paul Joseph Davis commented on COUCHDB-1367: @Randall If there's no technical reason why we need to bump the seq for changes that don't modify a document we could just stop doing that. What operations do this currently? How many of these resources have we at one time or another discussed making into a full document? I think _purge, _revs_limit, and _security. Maybe other things. Not sure about the strictly confirming to theoretical model for _revs_limit and _security, but _purge has obvious semantics that need to be updated in views. Though as I think about this, I'm fairly certain we have a bug in the indexer that no one has ever reported. Granted, that seems to be fairly true of things that touch _purge code. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175877#comment-13175877 ] Jason Smith commented on COUCHDB-1367: -- Wait a second. Robert, you are not fixing a bug in C-L, you are working around a deficiency in CouchDB. The only way to know the latest sequence id is to make a complete _changes query. Next, follow that up with a continuous feed if you want to keep the state fresh. That is a paper cut. What if I want to see the most recent five changes? What if there are a hundred million documents? What if 99% of the time, update_seq equals last_seq and so developers assume it means something it doesn't? Everybody wants to know the id of the latest change. Nobody wants to know the update sequence, whatever that is. If CouchDB can cache last_seq in the header and provide it in the DB response, that would be fantastic. Kindly reopen this ticket, then. Thanks! When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175892#comment-13175892 ] Randall Leeds commented on COUCHDB-1367: Wait a second. Robert, you are not fixing a bug in C-L, you are working around a deficiency in CouchDB. Can't both be true? The only way to know the latest sequence id is to make a complete _changes query. Next, follow that up with a continuous feed if you want to keep the state fresh. Nope. You can not ever know. You always know the latest sequence number at some arbitrarily recent point in time. If the (possibly continuous) changes feed says you're at X and you haven't heard anything more yet, then the database is at update sequence = X. Neveretheless, I think I follow the sentiment. If last_seq were there one could know from an info request whether or not more changes should be available. What if I want to see the most recent five changes? What if there are a hundred million documents? What if 99% of the time, update_seq equals last_seq and so developers assume it means something it doesn't? In order: * /_changes?descending=truelimit=5 * Not sure how this is relevant * This does indeed seem to cause some confusion. It clearly surprised Robert and Henrik and it's the first I've heard of this discrepancy. I submit that this bug is closed accurately with a suggestion to move proposals to improve the situation over to the dev list. Off the top of my head a partial list of suggestions goes something like: * Add additional information to the changes feed, perhaps with a query parameter (almost the reverse of include docs) * Stop incrementing the update sequence on certain kinds of non-document changes * Add more information to the db information response Please take it from there and we can work through a proposal. Thanks, everyone. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175357#comment-13175357 ] Robert Newson commented on COUCHDB-1367: On reflection, it's couchdb-lucene's bug, not couchdb's. Let me explain. CouchDB-Lucene (to give it its grown-up name) compares the update_seq from a GET /dbname to the sequences a background process is indexing through. It then unblocks searcher threads as that process reaches or exceeds the required update_seq. This is, in fact, just silly. Instead, a search query should cause a GET /dbname/_changes?since=latest index checkpoint. It should block until it consumes the entire response, passing the updates to the indexing process. It can then return a non-stale search result. In the case that the index is fresh, the _changes response contains no rows, and serves only to confirm that the index is fresh. If, as planned, CouchDB-Lucene *also* runs a _changes?feed=continuous to keep indexes fresh in the background then indexes will simply be fresher than they would be in the CouchDB case. I repeat, CouchDB-Lucene's *mistake* is to *only* use the feed=continuous variety of the changes feed. This prevents it from knowing when its own index is fresh. I will make this change next week and I suggest that this ticket be closed with no further action taken. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175367#comment-13175367 ] Bob Dionne commented on COUCHDB-1367: - wonderful, I wondering about this as this is precisely how I keep up to date in my native indexer, using _changes?since and storing the last_seq in a checkpoint. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175368#comment-13175368 ] Robert Newson commented on COUCHDB-1367: The motivation for c-l's current approach is for indexes to be fresh at all times as well as updating the lucene indexes optimally (in modest sized batches). If it were purely driven by user queries, it would be variably stale, and variably well optimized, like couchdb's indexes. I wanted better for my child. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175136#comment-13175136 ] Randall Leeds commented on COUCHDB-1367: I'm confused about why a client listening to continuous _changes should care whether or not update_seq changes without emitting a document modification. I like to think that in the future other sorts of changes might be allowed to surface in that feed. Is there any place we guarantee that seq in the changes feed should be monotonic? I don't think so. It seems to me like this is not a problem. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175143#comment-13175143 ] Robert Newson commented on COUCHDB-1367: I would hope it's obvious that update_seq must be monotonously incrementing (i.e, it cannot go down). You're right in the strict sense here, readers of the changes feed will get a row for each document change, and nothing else. The subtle point I think you've missed in that some applications want to know if they've read all changes up to the current update sequence of the database. non stale=ok view queries already do this, and couchdb-lucene does too. It turns out that *only* the view engine can do it correctly in all cases because it knows the last sequence value that affected a document (that is, it doesn't 'see' the change to _security, and thus doesn't block for that change). When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175160#comment-13175160 ] Randall Leeds commented on COUCHDB-1367: Oh, I messed up. Monotonic does not mean what I think it means. I meant it need not increase by intervals of 1 all the time. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175162#comment-13175162 ] Randall Leeds commented on COUCHDB-1367: I'd propose making the heartbeat option send the last_seq row, but that'd probably break clients that expect a disconnect at that point. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175168#comment-13175168 ] Robert Newson commented on COUCHDB-1367: Correct, 'monotonically incrementing' means that the number always goes up but does *not* imply that it always goes up by exactly 1. Because it sorta kind *sounds* like that, I clarified with 'never goes down' and tried a synonym for monotonic. The changes feed will certainly have gaps, but no row N will have a lower update_seq than any previously seen row. monotonic |ˌmänəˈtänik| adjective 1 Mathematics (of a function or quantity) varying in such a way that it either never decreases or never increases. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175217#comment-13175217 ] Randall Leeds commented on COUCHDB-1367: My gut says it's more confusing to put update_seq and last_seq in the db info. If clients want to be sure they're up to date, they need only pass a heartbeat parameter to the changes feed. If the heartbeats are coming in but no updates are, the feed is up to date. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175219#comment-13175219 ] Bob Dionne commented on COUCHDB-1367: - I agree, it would appear redundant to the user and confusing as it would most likely always be the same. I like the heartbeat solution, especially because it means we don't need to fix anything in couchdb When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[
https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175228#comment-13175228
]
Randall Leeds commented on COUCHDB-1367:
I suggested the heartbeat because we could make it look like last_seq, but we
don't even need to use heartbeat. We could emit a change that doesn't have a
corresponding id and revisions. Although, since the URL is /db/_revs_limit, we
could (to use that example) emit something like:
{seq:X,id:_revs_limit,changes:[]}
I have no idea how badly that confuse existing clients, including CouchDB.
Putting it on the db info is the least obtrusive from an API standpoint. From a
code internals, I think everything would require some change (with the
exception of doing nothing about this). I'm going to step away for now, but if
you need any more color swatches I can send over some more samples.
When settings revs_limit on db - the db increases its update_seq counter when
viewing stats - but not when getting changes
--
Key: COUCHDB-1367
URL: https://issues.apache.org/jira/browse/COUCHDB-1367
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.1.1
Environment: Any
Reporter: Henrik Hofmeister
Assignee: Bob Dionne
Priority: Minor
Labels: revs_limit
If you put a number to _revs_limit on a db (to update it) - the
http://host/dbname/ info document gets an increase in update_seq number -
however the changes feed does not contain this change (while its not a
change). This causes the update_seq in the dbinfo doc and the last seq in the
changes feed to differ - which breaks any application depending on the
update_seq number as the expected sequence size of the db (in my case -
couchdb-lucene that will only respond to stale requests because it thinks its
not up to date)
I know this is an edge case - but still its something fairly fundamental -
that clearly is not working as intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175236#comment-13175236 ] Jason Smith commented on COUCHDB-1367: -- I disagree with Randal's gut. The metadata from a /db response is not confusing anybody. Most people poke around for values they care about and ignore the rest. That committed_update_seq doesn't seem to bother anybody hopefully demonstrates this point. On the other hand, a new type of change object has more impact. For example, if I add ?include_docs=true, what would the .doc field be, if anything? So, in other words, IMO last_seq in the DB response is perfect. People have seen it in their _changes queries and they will make the right assumptions about its meanings. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175262#comment-13175262 ] Randall Leeds commented on COUCHDB-1367: For _revs_limit we could always surface a change to the document with id _revs_limit, which is _somewhat_ accurate (though it's not a full doc with revisions). Similar thoughts apply for _security. If there's no technical reason why we need to bump the seq for changes that don't modify a document we could just stop doing that. What operations do this currently? How many of these resources have we at one time or another discussed making into a full document? When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13175286#comment-13175286 ] Jason Smith commented on COUCHDB-1367: -- Special-case changes response remind me of the discussion local docs (I think WRT _security). It seems worth considering making couch have *fewer* special cases as it gains features. Could the revs limit be set in a _local/* document, which would have standard MVCC semantics (but they don't replicate)? Clients can examine and configure databases with their normal document manipulation functions, communicating with Couch through documents. The list of things that arguably belong in _local/ grows. The security object, and apparently now the revs limit value can still be stored in the file header, but that is only a cache. (Couch might even expose the legacy API and internally convert it to document updates.) Is it possible? When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173086#comment-13173086 ] Bob Dionne commented on COUCHDB-1367: - I'm a little confused then. It seems if c-l has access to the last_seq then it should be able to determine if there are no changes of interest. I thought the commentary on IRC was that this was not the case because it uses the continuous feed. Adding last_seq to the db info request will certainly solve it, though look a little redundant to the user as it will be almost always be equal to update_seq When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173289#comment-13173289 ] Henrik Hofmeister commented on COUCHDB-1367: What i'm puzzled about - is what would i ever need the update_seq for ? It allows me to - see that there has been made a change - however in the changes view it shows me that there are no changes? Only in the cases where it differs for last_seq of course - but what could i ever possibly use that number for? That is - a number - signalling that i have either updated revs_limit or a random other number of internal api calls ? Its absolutly useless - especially while i have no way of getting to know whats changed. update_seq would - in any possible case - be expected by the user to reflect your core feature - the changes feed? Not making it into a huge problem - but the only real fix for a production env. product like couchdb is to not add to the confusion - but fix the confusion (like not adding another number to the db info page) . That would give you 2 numbers - one that is useless (update_seq) and one that is the one you'd expect (last_seq). ? When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173539#comment-13173539 ] Dave Cottlehuber commented on COUCHDB-1367: --- +1 on last_update being consistent across replication, db, changes. I can't see anywhere else this is exposed via API. Now where to document that? When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172245#comment-13172245 ] Bob Dionne commented on COUCHDB-1367: - Henrik, Thanks for the report. We discussed this a bit on irc this morning. So last_seq in the changes feed and update_seq in the db info are not intended to be the same, or at least there's some confusion about the semantics. couchdb-lucene uses continuous changes feeds so it doesn't have access to the last_seq value of a normal changes feed. When update_seq changes due to a call to set_revs_limit it gets out of whack. In any event the solution may be to simply add last_seq to the db_info record. It shouldn't be hard to fix and as you say this is an edge case. I'm curious are you setting the revs_limit a lot? If so what's the use case? Bob When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172296#comment-13172296 ] Henrik Hofmeister commented on COUCHDB-1367: No im not - i just made a script ensuring it gets set on all dbs - to save a request i did the just set it to 1 approach - assuming it wouldn't matter if i set it to 1 several times. This is how i discovered there was such a problem. We had been getting it randomly before - without ever realizing what the problem was exactly. Its true that this specific case is for couchdb-lucene - however the general use case of being able to predict how far you're away from being up to date is not couchdb-lucene specific - i've for one created another in-house application that does exactly this - while performing a chained map-reduce like operation (What im saying is - if you want to reap the benefits of the changes feed and be aware of your progress - you'll need the right number) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172401#comment-13172401 ] Bob Dionne commented on COUCHDB-1367: - sure, I think the confusion is this (from the WIKI): last_seq is the sequence number of the last update returned. (Currently it will always be the same as the seq of the last item in results.) It doesn't say that this is identical to update_seq. They are two different things, or at least became that way when other features such as set_revs_limit were added. It might be better for now in your app to use last_seq if you can When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172432#comment-13172432 ] Paul Joseph Davis commented on COUCHDB-1367: Quite right. update_seq will be incremented for a number of API's that make changes while not updating documents. There's even an increment_update_seq API that just bumps it directly. Having a last_update in the db info blob that lists the last update for managing changes feeds might be useful in general and seems like it'd be an easy enough addition. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[
https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172752#comment-13172752
]
Jason Smith commented on COUCHDB-1367:
--
Hi, Bob. You said,
couchdb-lucene uses continuous changes feeds so it doesn't have access to the
last_seq value of a normal changes feed.
If the database is deleted during a continuous _changes request, CouchDB will
send {last_seq:N} and disconnect the client. Perhaps it's not relevant to
this bug. But a robust changes listener should be prepared to get a last_seq
message.
I learned this from a bug report in Follow
https://github.com/iriscouch/follow/issues/6
When settings revs_limit on db - the db increases its update_seq counter when
viewing stats - but not when getting changes
--
Key: COUCHDB-1367
URL: https://issues.apache.org/jira/browse/COUCHDB-1367
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.1.1
Environment: Any
Reporter: Henrik Hofmeister
Assignee: Bob Dionne
Priority: Minor
Labels: revs_limit
If you put a number to _revs_limit on a db (to update it) - the
http://host/dbname/ info document gets an increase in update_seq number -
however the changes feed does not contain this change (while its not a
change). This causes the update_seq in the dbinfo doc and the last seq in the
changes feed to differ - which breaks any application depending on the
update_seq number as the expected sequence size of the db (in my case -
couchdb-lucene that will only respond to stale requests because it thinks its
not up to date)
I know this is an edge case - but still its something fairly fundamental -
that clearly is not working as intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
[ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172791#comment-13172791 ] Robert Newson commented on COUCHDB-1367: Oh, ye of little faith. couchdb-lucene certainly can handle the last_seq output (and has since approximately forever). It's just so rarely encountered that it doesn't help with respect to the issue at hand. When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes -- Key: COUCHDB-1367 URL: https://issues.apache.org/jira/browse/COUCHDB-1367 Project: CouchDB Issue Type: Bug Components: HTTP Interface Affects Versions: 1.1.1 Environment: Any Reporter: Henrik Hofmeister Assignee: Bob Dionne Priority: Minor Labels: revs_limit If you put a number to _revs_limit on a db (to update it) - the http://host/dbname/ info document gets an increase in update_seq number - however the changes feed does not contain this change (while its not a change). This causes the update_seq in the dbinfo doc and the last seq in the changes feed to differ - which breaks any application depending on the update_seq number as the expected sequence size of the db (in my case - couchdb-lucene that will only respond to stale requests because it thinks its not up to date) I know this is an edge case - but still its something fairly fundamental - that clearly is not working as intended. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (COUCHDB-1367) When settings revs_limit on db - the db increases its update_seq counter when viewing stats - but not when getting changes
On Tue, Dec 20, 2011 at 7:39 AM, Robert Newson (Commented) (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/COUCHDB-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13172791#comment-13172791 ] Robert Newson commented on COUCHDB-1367: Oh, ye of little faith. couchdb-lucene certainly can handle the last_seq output (and has since approximately forever). It's just so rarely encountered that it doesn't help with respect to the issue at hand. First of all, faith is the most overrated virtue. It is, in fact, a vice. But no, sorry. I wasn't speaking about couchdb-lucene, but rather making a minor correction to Bob D.'s comment that a continuous listener hasn't access to the last_seq object. -- Iris Couch
