Hi Jan,
Following the implementation of your advice, I continued to see
inconsistent data. I mean your advice has probably helped on some updates
but not all.
So I continued to dig into the issue and finally got to the point where I
may have found some problem with 'byExample' (which is used to cache those
stats during the transaction).
Here is a snippet reproducing the behavior:
controller.post('/foxxdebug', function (req, res) {
var from = 'foxxdebug/' + makeid(32); // creates some random ID
var to = 'foxxdebug/' + makeid(32);
db._executeTransaction({
collections: {
read: ['foxxdebug'],
write: ['foxxdebug']
},
action: function () {
var newDoc = db.foxxdebug.insert(from, to, { request: true });
var fromCount1 = db.foxxdebug.byExample({ _from: from, request:
false }).count();
var toCount1 = db.foxxdebug.byExample({ _to: to, request: false
}).count();
newDoc.request = false;
db.foxxdebug.replace({ _id: newDoc._id }, newDoc);
var fromCount2 = db.foxxdebug.byExample({ _from: from, request:
false }).count();
var toCount2 = db.foxxdebug.byExample({ _to: to, request: false
}).count();
res.json({ fromCount1: fromCount1, toCount1: toCount1,
fromCount2: fromCount2, toCount2: toCount2 });
}
});
});
I would expect the response to be:
{ "fromCount1": 0, "toCount1": 0, "fromCount2": 1, "toCount2": 1 }
but I get
{ "fromCount1": 1, "toCount1": 1, "fromCount2": 1, "toCount2": 1 }
and I'm pretty sure that's related to the issue I'm seeing.
Any further help would be greatly appreciated, thanks!
On Tuesday, June 14, 2016 at 11:32:37 PM UTC+8, Jan Steemann wrote:
>
> Hi Thomas,
>
> yes, I think that should also fix it.
> `post.stats = ...` will set the `stats` property of the `post` object,
> which is copy-on-write and thus fine.
> Only accessing a sub-property of a property will cause issues when the
> document is a ShapedJson instance.
>
> Best regards
> Jan
>
> 2016-06-14 15:27 GMT+02:00 Thomas Weiss <[email protected]
> <javascript:>>:
>
>> Hi Jan,
>>
>> Thank you so much for this very extensive reply. I know you guys are
>> working hard on the 3.0 release so I really appreciate the time you took to
>> write this answer!
>>
>> If I understand correctly what you explained, am I right that the
>> following code would also solve my issue:
>> post.stats = {
>> likeCount: db.likes.byExample({ _to: 'posts/' + postId }).count(),
>> commentCount: post.stats.commentCount,
>> shareCount: post.stats.shareCount
>> };
>> db.posts.replace({ _id: post._id }, post);
>>
>> Cheers,
>> Thomas
>>
>> On Tuesday, June 14, 2016 at 3:59:57 PM UTC+8, Jan Steemann wrote:
>>>
>>> Hi there,
>>>
>>> I think I can shed some light on this. I don't think this is related to
>>> waitForSync or transaction, but has a different root cause.
>>>
>>> All documents, when inserted, updated or replaced are saved in
>>> ArangoDB's write-ahead log (WAL) first.
>>> When retrieving a document from the WAL using
>>> db.<collection>.document(key) then a copy of the document will be returned
>>> to Foxx/JavaScript as a regular JavaScript object:
>>>
>>> /* fetch document with key postId from database. will return a
>>> JavaScript object */
>>> db.posts.insert({ _key: postId, stats: { likeCount: 0 } });
>>> var post = db.posts.document(postId);
>>>
>>> This object can be modified regularly and saved again, using
>>> update/replace:
>>>
>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>> }).count();
>>> db.posts.replace({ _id: post._id }, post);
>>>
>>> However, at some point the documents are moved from the WAL into the
>>> datafiles of the respective collections.
>>> When they are then retrieved from the datafiles, they are returned as
>>> light-weight JavaScript objects which contain only pointers to the
>>> underlying document data.
>>> Accessing a property of such light-weight object will trigger a property
>>> handler, which will build the result value and return it as a *temporary
>>> object*
>>> That means when accessing `post.stats` in the following code,
>>> `post.stats` will return a temporary object, for which the property
>>> `likeCount` will be updated:
>>>
>>> /* returns light-weight object with property handlers */
>>> var post = db.posts.document(postId);
>>> /* post.stats will produce a tempory object, and
>>> post.state.likeCount will access a property of the temporary object */
>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>> }).count();
>>>
>>> The modification will only happen in the temporary object, but not in
>>> the `post` object.
>>> When then the post object is written back to the database, it won't
>>> contain the changes.
>>>
>>> Following is some example code that demonstrates this for a document
>>> that is stored in the WAL and for a document from the collection datafiles.
>>> In one case the update/replace will do what's expected and in the other
>>> it won't:
>>>
>>> ```
>>> db._drop("posts");
>>> db._drop("likes");
>>>
>>> db._create("posts");
>>> db._createEdgeCollection("likes");
>>>
>>> var ShapedJson = require("internal").ShapedJson;
>>>
>>> function updateCount(issuerId, postId) {
>>> var post = db.posts.document(postId);
>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true);
>>> // the post object was fetched before
>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId
>>> }).count();
>>> db.posts.replace({ _id: post._id }, post);
>>> }
>>>
>>> // insert a new post document. it will be located in the write-ahead log
>>> first
>>> db.posts.insert({ _key: "post1", stats: { likeCount: 0 } });
>>> require("internal").print("the document:", db.posts.document("post1"));
>>> require("internal").print("is ShapedJson:", db.posts.document("post1")
>>> instanceof ShapedJson);
>>>
>>> // update the document's count. this will work fine
>>> updateCount("test1", "post1");
>>>
>>> // and print results. count should have been updated
>>> require("internal").print("count for post1 after update is:",
>>> db.posts.document("post1"));
>>>
>>>
>>>
>>> // insert another post document. it will be located in the write-ahead
>>> log first
>>> db.posts.insert({ _key: "post2", stats: { likeCount: 0 } });
>>> // now flush the write-ahead log manually and wait for a few seconds
>>> // note: flushing the write-ahead log may happen automatically from time
>>> to time while
>>> // the server is running
>>> require("internal").wal.flush(true, true);
>>> require("internal").wait(5, false);
>>>
>>> require("internal").print("the document:", db.posts.document("post2"));
>>> require("internal").print("is ShapedJson:", db.posts.document("post2")
>>> instanceof ShapedJson);
>>>
>>> // now update the document's count. this will update the count value of
>>> a ShapedJson
>>> // (which is a temporary object when retrieved from the database)
>>> updateCount("test2", "post2");
>>>
>>> // and print results. count will not have been updated
>>> require("internal").print("count for post2 is now:",
>>> db.posts.document("post2"));
>>> ```
>>>
>>> Note that in the above example for both documents there is a check
>>> whether they are an instance of *ShapedJson*.
>>> In the one case (WAL case) the document won't be a ShapedJson instance
>>> but a regular JS object, but in the other case it will be.
>>> In the latter case, updating a property of the document which itself is
>>> an array or object won't work, because of the temporaries produced by the
>>> property handler.
>>>
>>> What helps in this case is to convert the ShapedJson instance of the
>>> document into a regular JS object, e.g. via
>>>
>>> var _ = require("underscore");
>>> ...
>>> var post = _.clone(db.posts.document(postId));
>>>
>>> Or, combined with the ShapedJson test:
>>>
>>> var ShapedJson = require("internal").ShapedJson;
>>> var _ = require("underscore");
>>> ...
>>> var post = db.posts.document(postId);
>>>
>>>
>>> if (post instance of ShapedJson) {
>>> // ShapedJson object
>>> post = _.clone(post);
>>> }
>>>
>>> Note that this applies to the 1.x and 2.x branches of ArangoDB.
>>> We have simplified this a lot in 3.0 so both the ShapedJson testing and
>>> cloning can be omitted there. All objects returned in 3.0 will be regular
>>> JS objects free of side effects.
>>>
>>> I hope this helps.
>>> Best regards
>>> Jan
>>>
>>>
>>>
>>> 2016-06-14 5:18 GMT+02:00 Thomas Weiss <[email protected]>:
>>>
>>>> As a follow-up, I've just seen the same behavior on my dev machine, it
>>>> just happens less frequently (probably because it has better perfs than
>>>> the
>>>> server used for staging).
>>>> Any comment will be welcome!
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Sunday, June 12, 2016 at 3:57:04 PM UTC+8, Thomas Weiss wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> I'm using a Foxx app to execute transactions. To boost query
>>>>> efficiency (the system is read-heavy), I try to cache the count of edges
>>>>> directly in the docs, and do this like that:
>>>>> db.likes.insert('users/' + issuerId, 'posts/' + postId, {}, true);
>>>>> // the post object was fetched before
>>>>> post.stats.likeCount = db.likes.byExample({ _to: 'posts/' + postId }).
>>>>> count();
>>>>> db.posts.replace({ _id: post._id }, post);
>>>>> Notice the last *true* parameter passed to *insert* to make sure that
>>>>> we wait for the write to be flushed. This code is executed within a
>>>>> transaction by the way.
>>>>>
>>>>> This works well on my dev machine (Windows 10) *but* it seems that
>>>>> the count is not systematically updated correctly on the staging server
>>>>> (Ubuntu). What's weird is that sometimes it works and sometimes it
>>>>> doesn't,
>>>>> which makes me think about a threading/concurrency issue (and that's why
>>>>> I
>>>>> added the waitForSync param).
>>>>>
>>>>> Any help or suggestion would be greatly appreciated here.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "ArangoDB" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "ArangoDB" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"ArangoDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
