By the way, the workaround to prevent the issue from occurring is to not use `byExample(...).count()` but instead use `byExample(...).toArray().length`. We plan to build a new 2.8 release this week end.
Best regards Jan 2016-06-28 11:54 GMT+02:00 Jan Steemann <[email protected]>: > Hi Thomas, > > I used your code to reproduce the issue and it's partly related. > The problem in this case is not that there are immutable objects, but that > the call to `byExample(...).count()` returns an unexpected value. > > var fromCount1 = db.foxxdebug.byExample({ _from: from, request: false }). > count(); > > The `byExample()` returns an object of type `SimpleQueryByExample`, and > when calling `toArray()` on this, the results are correct, before and after > the update. > However, when calling `count()` on that object, this will also correctly > execute the simple query, but sometimes returns a wrong result for `count`. > The reason for this is that internally the result is produced by an index > lookup and then may need to be post-filtered in order to return only those > documents that match all conditions. In this case, the byExample will use > the edge index on `_from` and then post-filter the result using the > `request == false` condition. The result of post-filtering is also correct, > however, the `count` value of the query is not adjusted. `count` in this > case will return the number of documents before post-filtering. > > In your case the number of documents with the queried `_from` and `_to` > values don't change due to the replace, so the `count` values before and > after the replace are identical. Clearly it's a bug that the count value is > wrong, and I just fixed it in the 2.8 branch. I checked that it's already > working fine in 3.0, and 2.8 is the last affected version. > > Best regards > Jan > > > 2016-06-25 6:12 GMT+02:00 Thomas Weiss <[email protected]>: > >> Hi Jan, >> >> Following the implementation of your advice, I continued to see >> inconsistent data. I mean your advice has probably helped on some updates >> but not all. >> So I continued to dig into the issue and finally got to the point where I >> may have found some problem with 'byExample' (which is used to cache those >> stats during the transaction). >> >> Here is a snippet reproducing the behavior: >> controller.post('/foxxdebug', function (req, res) { >> var from = 'foxxdebug/' + makeid(32); // creates some random ID >> var to = 'foxxdebug/' + makeid(32); >> db._executeTransaction({ >> collections: { >> read: ['foxxdebug'], >> write: ['foxxdebug'] >> }, >> action: function () { >> var newDoc = db.foxxdebug.insert(from, to, { request: true >> }); >> var fromCount1 = db.foxxdebug.byExample({ _from: from, >> request: false }).count(); >> var toCount1 = db.foxxdebug.byExample({ _to: to, request: >> false }).count(); >> >> newDoc.request = false; >> db.foxxdebug.replace({ _id: newDoc._id }, newDoc); >> var fromCount2 = db.foxxdebug.byExample({ _from: from, >> request: false }).count(); >> var toCount2 = db.foxxdebug.byExample({ _to: to, request: >> false }).count(); >> >> res.json({ fromCount1: fromCount1, toCount1: toCount1, >> fromCount2: fromCount2, toCount2: toCount2 }); >> } >> }); >> }); >> >> >> I would expect the response to be: >> { "fromCount1": 0, "toCount1": 0, "fromCount2": 1, "toCount2": 1 } >> but I get >> { "fromCount1": 1, "toCount1": 1, "fromCount2": 1, "toCount2": 1 } >> and I'm pretty sure that's related to the issue I'm seeing. >> >> Any further help would be greatly appreciated, thanks! >> >> On Tuesday, June 14, 2016 at 11:32:37 PM UTC+8, Jan Steemann wrote: >>> >>> Hi Thomas, >>> >>> yes, I think that should also fix it. >>> `post.stats = ...` will set the `stats` property of the `post` object, >>> which is copy-on-write and thus fine. >>> Only accessing a sub-property of a property will cause issues when the >>> document is a ShapedJson instance. >>> >>> Best regards >>> Jan >>> >>> 2016-06-14 15:27 GMT+02:00 Thomas Weiss <[email protected]>: >>> >>>> Hi Jan, >>>> >>>> Thank you so much for this very extensive reply. I know you guys are >>>> working hard on the 3.0 release so I really appreciate the time you took to >>>> write this answer! >>>> >>>> If I understand correctly what you explained, am I right that the >>>> following code would also solve my issue: >>>> post.stats = { >>>> likeCount: db.likes.byExample({ _to: 'posts/' + postId }).count(), >>>> commentCount: post.stats.commentCount, >>>> shareCount: post.stats.shareCount >>>> }; >>>> db.posts.replace({ _id: post._id }, post); >>>> >>>> Cheers, >>>> Thomas >>>> >>>> On Tuesday, June 14, 2016 at 3:59:57 PM UTC+8, Jan Steemann wrote: >>>>> >>>>> Hi there, >>>>> >>>>> I think I can shed some light on this. I don't think this is related >>>>> to waitForSync or transaction, but has a different root cause. >>>>> >>>>> All documents, when inserted, updated or replaced are saved in >>>>> ArangoDB's write-ahead log (WAL) first. >>>>> When retrieving a document from the WAL using >>>>> db.<collection>.document(key) then a copy of the document will be returned >>>>> to Foxx/JavaScript as a regular JavaScript object: >>>>> >>>>> /* fetch document with key postId from database. will return a >>>>> JavaScript object */ >>>>> db.posts.insert({ _key: postId, stats: { likeCount: 0 } }); >>>>> var post = db.posts.document(postId); >>>>> >>>>> This object can be modified regularly and saved again, using >>>>> update/replace: >>>>> >>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId >>>>> }).count(); >>>>> db.posts.replace({ _id: post._id }, post); >>>>> >>>>> However, at some point the documents are moved from the WAL into the >>>>> datafiles of the respective collections. >>>>> When they are then retrieved from the datafiles, they are returned as >>>>> light-weight JavaScript objects which contain only pointers to the >>>>> underlying document data. >>>>> Accessing a property of such light-weight object will trigger a >>>>> property handler, which will build the result value and return it as a >>>>> *temporary object* >>>>> That means when accessing `post.stats` in the following code, >>>>> `post.stats` will return a temporary object, for which the property >>>>> `likeCount` will be updated: >>>>> >>>>> /* returns light-weight object with property handlers */ >>>>> var post = db.posts.document(postId); >>>>> /* post.stats will produce a tempory object, and >>>>> post.state.likeCount will access a property of the temporary object */ >>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId >>>>> }).count(); >>>>> >>>>> The modification will only happen in the temporary object, but not in >>>>> the `post` object. >>>>> When then the post object is written back to the database, it won't >>>>> contain the changes. >>>>> >>>>> Following is some example code that demonstrates this for a document >>>>> that is stored in the WAL and for a document from the collection >>>>> datafiles. >>>>> In one case the update/replace will do what's expected and in the >>>>> other it won't: >>>>> >>>>> ``` >>>>> db._drop("posts"); >>>>> db._drop("likes"); >>>>> >>>>> db._create("posts"); >>>>> db._createEdgeCollection("likes"); >>>>> >>>>> var ShapedJson = require("internal").ShapedJson; >>>>> >>>>> function updateCount(issuerId, postId) { >>>>> var post = db.posts.document(postId); >>>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true); >>>>> // the post object was fetched before >>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId >>>>> }).count(); >>>>> db.posts.replace({ _id: post._id }, post); >>>>> } >>>>> >>>>> // insert a new post document. it will be located in the write-ahead >>>>> log first >>>>> db.posts.insert({ _key: "post1", stats: { likeCount: 0 } }); >>>>> require("internal").print("the document:", db.posts.document("post1")); >>>>> require("internal").print("is ShapedJson:", db.posts.document("post1") >>>>> instanceof ShapedJson); >>>>> >>>>> // update the document's count. this will work fine >>>>> updateCount("test1", "post1"); >>>>> >>>>> // and print results. count should have been updated >>>>> require("internal").print("count for post1 after update is:", >>>>> db.posts.document("post1")); >>>>> >>>>> >>>>> >>>>> // insert another post document. it will be located in the write-ahead >>>>> log first >>>>> db.posts.insert({ _key: "post2", stats: { likeCount: 0 } }); >>>>> // now flush the write-ahead log manually and wait for a few seconds >>>>> // note: flushing the write-ahead log may happen automatically from >>>>> time to time while >>>>> // the server is running >>>>> require("internal").wal.flush(true, true); >>>>> require("internal").wait(5, false); >>>>> >>>>> require("internal").print("the document:", db.posts.document("post2")); >>>>> require("internal").print("is ShapedJson:", db.posts.document("post2") >>>>> instanceof ShapedJson); >>>>> >>>>> // now update the document's count. this will update the count value >>>>> of a ShapedJson >>>>> // (which is a temporary object when retrieved from the database) >>>>> updateCount("test2", "post2"); >>>>> >>>>> // and print results. count will not have been updated >>>>> require("internal").print("count for post2 is now:", >>>>> db.posts.document("post2")); >>>>> ``` >>>>> >>>>> Note that in the above example for both documents there is a check >>>>> whether they are an instance of *ShapedJson*. >>>>> In the one case (WAL case) the document won't be a ShapedJson instance >>>>> but a regular JS object, but in the other case it will be. >>>>> In the latter case, updating a property of the document which itself >>>>> is an array or object won't work, because of the temporaries produced by >>>>> the property handler. >>>>> >>>>> What helps in this case is to convert the ShapedJson instance of the >>>>> document into a regular JS object, e.g. via >>>>> >>>>> var _ = require("underscore"); >>>>> ... >>>>> var post = _.clone(db.posts.document(postId)); >>>>> >>>>> Or, combined with the ShapedJson test: >>>>> >>>>> var ShapedJson = require("internal").ShapedJson; >>>>> var _ = require("underscore"); >>>>> ... >>>>> var post = db.posts.document(postId); >>>>> >>>>> >>>>> if (post instance of ShapedJson) { >>>>> // ShapedJson object >>>>> post = _.clone(post); >>>>> } >>>>> >>>>> Note that this applies to the 1.x and 2.x branches of ArangoDB. >>>>> We have simplified this a lot in 3.0 so both the ShapedJson testing >>>>> and cloning can be omitted there. All objects returned in 3.0 will be >>>>> regular JS objects free of side effects. >>>>> >>>>> I hope this helps. >>>>> Best regards >>>>> Jan >>>>> >>>>> >>>>> >>>>> 2016-06-14 5:18 GMT+02:00 Thomas Weiss <[email protected]>: >>>>> >>>>>> As a follow-up, I've just seen the same behavior on my dev machine, >>>>>> it just happens less frequently (probably because it has better perfs >>>>>> than >>>>>> the server used for staging). >>>>>> Any comment will be welcome! >>>>>> >>>>>> Thanks, >>>>>> Thomas >>>>>> >>>>>> >>>>>> On Sunday, June 12, 2016 at 3:57:04 PM UTC+8, Thomas Weiss wrote: >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>> I'm using a Foxx app to execute transactions. To boost query >>>>>>> efficiency (the system is read-heavy), I try to cache the count of edges >>>>>>> directly in the docs, and do this like that: >>>>>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true); >>>>>>> // the post object was fetched before >>>>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId >>>>>>> }).count(); >>>>>>> db.posts.replace({ _id: post._id }, post); >>>>>>> Notice the last *true* parameter passed to *insert* to make sure >>>>>>> that we wait for the write to be flushed. This code is executed within a >>>>>>> transaction by the way. >>>>>>> >>>>>>> This works well on my dev machine (Windows 10) *but* it seems that >>>>>>> the count is not systematically updated correctly on the staging server >>>>>>> (Ubuntu). What's weird is that sometimes it works and sometimes it >>>>>>> doesn't, >>>>>>> which makes me think about a threading/concurrency issue (and that's >>>>>>> why I >>>>>>> added the waitForSync param). >>>>>>> >>>>>>> Any help or suggestion would be greatly appreciated here. >>>>>>> >>>>>>> Thanks, >>>>>>> Thomas >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ArangoDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "ArangoDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "ArangoDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "ArangoDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
