Hi Thomas,
I used your code to reproduce the issue and it's partly related.
The problem in this case is not that there are immutable objects, but that
the call to `byExample(...).count()` returns an unexpected value.
var fromCount1 = db.foxxdebug.byExample({ _from: from, request: false }).
count();
The `byExample()` returns an object of type `SimpleQueryByExample`, and
when calling `toArray()` on this, the results are correct, before and after
the update.
However, when calling `count()` on that object, this will also correctly
execute the simple query, but sometimes returns a wrong result for `count`.
The reason for this is that internally the result is produced by an index
lookup and then may need to be post-filtered in order to return only those
documents that match all conditions. In this case, the byExample will use
the edge index on `_from` and then post-filter the result using the
`request == false` condition. The result of post-filtering is also correct,
however, the `count` value of the query is not adjusted. `count` in this
case will return the number of documents before post-filtering.
In your case the number of documents with the queried `_from` and `_to`
values don't change due to the replace, so the `count` values before and
after the replace are identical. Clearly it's a bug that the count value is
wrong, and I just fixed it in the 2.8 branch. I checked that it's already
working fine in 3.0, and 2.8 is the last affected version.
By the way, the workaround to prevent the issue from occurring is to not
use `byExample(...).count()` but instead use
`byExample(...).toArray().length`.
We plan to build a new 2.8 release this week end.
Best regards
Jan
2016-06-25 6:12 GMT+02:00 Thomas Weiss <[email protected]>:
> Hi Jan,
>
> Following the implementation of your advice, I continued to see
> inconsistent data. I mean your advice has probably helped on some updates
> but not all.
> So I continued to dig into the issue and finally got to the point where I
> may have found some problem with 'byExample' (which is used to cache those
> stats during the transaction).
>
> Here is a snippet reproducing the behavior:
> controller.post('/foxxdebug', function (req, res) {
> var from = 'foxxdebug/' + makeid(32); // creates some random ID
> var to = 'foxxdebug/' + makeid(32);
> db._executeTransaction({
> collections: {
> read: ['foxxdebug'],
> write: ['foxxdebug']
> },
> action: function () {
> var newDoc = db.foxxdebug.insert(from, to, { request: true });
> var fromCount1 = db.foxxdebug.byExample({ _from: from, request
> : false }).count();
> var toCount1 = db.foxxdebug.byExample({ _to: to, request:
> false }).count();
>
> newDoc.request = false;
> db.foxxdebug.replace({ _id: newDoc._id }, newDoc);
> var fromCount2 = db.foxxdebug.byExample({ _from: from, request
> : false }).count();
> var toCount2 = db.foxxdebug.byExample({ _to: to, request:
> false }).count();
>
> res.json({ fromCount1: fromCount1, toCount1: toCount1,
> fromCount2: fromCount2, toCount2: toCount2 });
> }
> });
> });
>
>
> I would expect the response to be:
> { "fromCount1": 0, "toCount1": 0, "fromCount2": 1, "toCount2": 1 }
> but I get
> { "fromCount1": 1, "toCount1": 1, "fromCount2": 1, "toCount2": 1 }
> and I'm pretty sure that's related to the issue I'm seeing.
>
> Any further help would be greatly appreciated, thanks!
>
> On Tuesday, June 14, 2016 at 11:32:37 PM UTC+8, Jan Steemann wrote:
>>
>> Hi Thomas,
>>
>> yes, I think that should also fix it.
>> `post.stats = ...` will set the `stats` property of the `post` object,
>> which is copy-on-write and thus fine.
>> Only accessing a sub-property of a property will cause issues when the
>> document is a ShapedJson instance.
>>
>> Best regards
>> Jan
>>
>> 2016-06-14 15:27 GMT+02:00 Thomas Weiss <[email protected]>:
>>
>>> Hi Jan,
>>>
>>> Thank you so much for this very extensive reply. I know you guys are
>>> working hard on the 3.0 release so I really appreciate the time you took to
>>> write this answer!
>>>
>>> If I understand correctly what you explained, am I right that the
>>> following code would also solve my issue:
>>> post.stats = {
>>> likeCount: db.likes.byExample({ _to: 'posts/' + postId }).count(),
>>> commentCount: post.stats.commentCount,
>>> shareCount: post.stats.shareCount
>>> };
>>> db.posts.replace({ _id: post._id }, post);
>>>
>>> Cheers,
>>> Thomas
>>>
>>> On Tuesday, June 14, 2016 at 3:59:57 PM UTC+8, Jan Steemann wrote:
>>>>
>>>> Hi there,
>>>>
>>>> I think I can shed some light on this. I don't think this is related to
>>>> waitForSync or transaction, but has a different root cause.
>>>>
>>>> All documents, when inserted, updated or replaced are saved in
>>>> ArangoDB's write-ahead log (WAL) first.
>>>> When retrieving a document from the WAL using
>>>> db.<collection>.document(key) then a copy of the document will be returned
>>>> to Foxx/JavaScript as a regular JavaScript object:
>>>>
>>>> /* fetch document with key postId from database. will return a
>>>> JavaScript object */
>>>> db.posts.insert({ _key: postId, stats: { likeCount: 0 } });
>>>> var post = db.posts.document(postId);
>>>>
>>>> This object can be modified regularly and saved again, using
>>>> update/replace:
>>>>
>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>>> }).count();
>>>> db.posts.replace({ _id: post._id }, post);
>>>>
>>>> However, at some point the documents are moved from the WAL into the
>>>> datafiles of the respective collections.
>>>> When they are then retrieved from the datafiles, they are returned as
>>>> light-weight JavaScript objects which contain only pointers to the
>>>> underlying document data.
>>>> Accessing a property of such light-weight object will trigger a
>>>> property handler, which will build the result value and return it as a
>>>> *temporary object*
>>>> That means when accessing `post.stats` in the following code,
>>>> `post.stats` will return a temporary object, for which the property
>>>> `likeCount` will be updated:
>>>>
>>>> /* returns light-weight object with property handlers */
>>>> var post = db.posts.document(postId);
>>>> /* post.stats will produce a tempory object, and
>>>> post.state.likeCount will access a property of the temporary object */
>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>>> }).count();
>>>>
>>>> The modification will only happen in the temporary object, but not in
>>>> the `post` object.
>>>> When then the post object is written back to the database, it won't
>>>> contain the changes.
>>>>
>>>> Following is some example code that demonstrates this for a document
>>>> that is stored in the WAL and for a document from the collection datafiles.
>>>> In one case the update/replace will do what's expected and in the other
>>>> it won't:
>>>>
>>>> ```
>>>> db._drop("posts");
>>>> db._drop("likes");
>>>>
>>>> db._create("posts");
>>>> db._createEdgeCollection("likes");
>>>>
>>>> var ShapedJson = require("internal").ShapedJson;
>>>>
>>>> function updateCount(issuerId, postId) {
>>>> var post = db.posts.document(postId);
>>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true);
>>>> // the post object was fetched before
>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>>> }).count();
>>>> db.posts.replace({ _id: post._id }, post);
>>>> }
>>>>
>>>> // insert a new post document. it will be located in the write-ahead
>>>> log first
>>>> db.posts.insert({ _key: "post1", stats: { likeCount: 0 } });
>>>> require("internal").print("the document:", db.posts.document("post1"));
>>>> require("internal").print("is ShapedJson:", db.posts.document("post1")
>>>> instanceof ShapedJson);
>>>>
>>>> // update the document's count. this will work fine
>>>> updateCount("test1", "post1");
>>>>
>>>> // and print results. count should have been updated
>>>> require("internal").print("count for post1 after update is:",
>>>> db.posts.document("post1"));
>>>>
>>>>
>>>>
>>>> // insert another post document. it will be located in the write-ahead
>>>> log first
>>>> db.posts.insert({ _key: "post2", stats: { likeCount: 0 } });
>>>> // now flush the write-ahead log manually and wait for a few seconds
>>>> // note: flushing the write-ahead log may happen automatically from
>>>> time to time while
>>>> // the server is running
>>>> require("internal").wal.flush(true, true);
>>>> require("internal").wait(5, false);
>>>>
>>>> require("internal").print("the document:", db.posts.document("post2"));
>>>> require("internal").print("is ShapedJson:", db.posts.document("post2")
>>>> instanceof ShapedJson);
>>>>
>>>> // now update the document's count. this will update the count value of
>>>> a ShapedJson
>>>> // (which is a temporary object when retrieved from the database)
>>>> updateCount("test2", "post2");
>>>>
>>>> // and print results. count will not have been updated
>>>> require("internal").print("count for post2 is now:",
>>>> db.posts.document("post2"));
>>>> ```
>>>>
>>>> Note that in the above example for both documents there is a check
>>>> whether they are an instance of *ShapedJson*.
>>>> In the one case (WAL case) the document won't be a ShapedJson instance
>>>> but a regular JS object, but in the other case it will be.
>>>> In the latter case, updating a property of the document which itself is
>>>> an array or object won't work, because of the temporaries produced by the
>>>> property handler.
>>>>
>>>> What helps in this case is to convert the ShapedJson instance of the
>>>> document into a regular JS object, e.g. via
>>>>
>>>> var _ = require("underscore");
>>>> ...
>>>> var post = _.clone(db.posts.document(postId));
>>>>
>>>> Or, combined with the ShapedJson test:
>>>>
>>>> var ShapedJson = require("internal").ShapedJson;
>>>> var _ = require("underscore");
>>>> ...
>>>> var post = db.posts.document(postId);
>>>>
>>>>
>>>> if (post instance of ShapedJson) {
>>>> // ShapedJson object
>>>> post = _.clone(post);
>>>> }
>>>>
>>>> Note that this applies to the 1.x and 2.x branches of ArangoDB.
>>>> We have simplified this a lot in 3.0 so both the ShapedJson testing and
>>>> cloning can be omitted there. All objects returned in 3.0 will be regular
>>>> JS objects free of side effects.
>>>>
>>>> I hope this helps.
>>>> Best regards
>>>> Jan
>>>>
>>>>
>>>>
>>>> 2016-06-14 5:18 GMT+02:00 Thomas Weiss <[email protected]>:
>>>>
>>>>> As a follow-up, I've just seen the same behavior on my dev machine, it
>>>>> just happens less frequently (probably because it has better perfs than
>>>>> the
>>>>> server used for staging).
>>>>> Any comment will be welcome!
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Sunday, June 12, 2016 at 3:57:04 PM UTC+8, Thomas Weiss wrote:
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I'm using a Foxx app to execute transactions. To boost query
>>>>>> efficiency (the system is read-heavy), I try to cache the count of edges
>>>>>> directly in the docs, and do this like that:
>>>>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true);
>>>>>> // the post object was fetched before
>>>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>>>>> }).count();
>>>>>> db.posts.replace({ _id: post._id }, post);
>>>>>> Notice the last *true* parameter passed to *insert* to make sure
>>>>>> that we wait for the write to be flushed. This code is executed within a
>>>>>> transaction by the way.
>>>>>>
>>>>>> This works well on my dev machine (Windows 10) *but* it seems that
>>>>>> the count is not systematically updated correctly on the staging server
>>>>>> (Ubuntu). What's weird is that sometimes it works and sometimes it
>>>>>> doesn't,
>>>>>> which makes me think about a threading/concurrency issue (and that's why
>>>>>> I
>>>>>> added the waitForSync param).
>>>>>>
>>>>>> Any help or suggestion would be greatly appreciated here.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "ArangoDB" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "ArangoDB" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "ArangoDB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"ArangoDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
