答复: 回复：Re: 答复: Cache.invoke are terribly slow and can't update cache
Thanks Val, that's good. -邮件原件- 发件人: vkulichenko [mailto:valentin.kuliche...@gmail.com] 发送时间: 2016年12月23日 9:47 收件人: user@ignite.apache.org 主题: Re: 回复：Re: 答复: Cache.invoke are terribly slow and can't update cache Shawn, You just should keep in mind that this is an object that will be serialized, sent across network and invoked on other side. The purpose of entry processor is to be executed on server side and atomically read and update a single entry. Having said that, the logic you described doesn't make much sense there. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slo w-and-can-t-update-cache-tp9676p9712.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 回复：Re: 答复: Cache.invoke are terribly slow and can't update cache
Shawn, You just should keep in mind that this is an object that will be serialized, sent across network and invoked on other side. The purpose of entry processor is to be executed on server side and atomically read and update a single entry. Having said that, the logic you described doesn't make much sense there. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slow-and-can-t-update-cache-tp9676p9712.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
回复：Re: 答复: Cache.invoke are terribly slow and can't update cache
Thanks Val!Now I understood entryprocessor better. as to "error prone", I meant if we follow the document,it will be error prone. For EntryProcessor is just an interface, it doesn't prohibit user to use lamda or anonymous classes. When we implement the interface at client side, the user may use client side global variables/classes/instances(some of them even will start new threads), I don't know it will work or not, at least it will introduce much complexity. At yesterday's debuging, I event don't see much useful logs. back to the document, I think it can be improved and give some recommendations. ThanksShawn 在2016年12月23日 03:03， vkulichenko 写道： Shawn, #1 - Yes, peer class loading is needed if you don't have entry processor implementation class deployed on server nodes. #2 - Correct. Moreover, I would recommend not to use lambdas or anonymous classes for anything that can be serialized, but to use static classes instead (this is actually a recommendation for Java in general, not only for Ignite). This will give you much more control on what is serialized along with entry processor and will simplify the understanding of how it is executed. Can you clarify why do you think it's error prone? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slow-and-can-t-update-cache-tp9676p9708.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 答复: Cache.invoke are terribly slow and can't update cache
Shawn, #1 - Yes, peer class loading is needed if you don't have entry processor implementation class deployed on server nodes. #2 - Correct. Moreover, I would recommend not to use lambdas or anonymous classes for anything that can be serialized, but to use static classes instead (this is actually a recommendation for Java in general, not only for Ignite). This will give you much more control on what is serialized along with entry processor and will simplify the understanding of how it is executed. Can you clarify why do you think it's error prone? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slow-and-can-t-update-cache-tp9676p9708.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
答复: Cache.invoke are terribly slow and can't update cache
Hi, I fix it by myself. I make two mistakes. #1 peerClassLoadingEnabled is disabled by default. Should enable it. #2 get ignite instance using: @IgniteInstanceResource Ignite ignite; Can't pass client ignite instance to it. Anyway, IMO, for EntryProcessor API it is a little error prone. Thanks Shawn -邮件原件- 发件人: Shawn Du [mailto:shawn...@neulion.com.cn] 发送时间: 2016年12月22日 10:34 收件人: user@ignite.apache.org 主题: 答复: Cache.invoke are terribly slow and can't update cache Thanks Val! Can you explain more detail how "invoke" works internally comparing with put operation. For we pass a "function" to ignite. I see below Java Doc about "invoke" * An instance of entry processor must be stateless as it may be invoked multiple times on primary and * backup nodes in the cache. It is guaranteed that the value passed to the entry processor will be always * the same. My question: can we use *outer* object in the function process of EntryProcessor which is visible in its scope? More complex, how to iterate, just like in my code? Thanks Shawn -邮件原件- 发件人: vkulichenko [mailto:valentin.kuliche...@gmail.com] 发送时间: 2016年12月22日 2:52 收件人: user@ignite.apache.org 主题: Re: Cache.invoke are terribly slow and can't update cache Hi Shawn, Cache is not updated because you never update it :) i.e. entry.setValue() is never called. Bad performance can be caused by the fact that you do multiple operations one after another in a single thread without any batching. Consider using invokeAll or IgniteDataStreamer. As for OOME, it's hard to tell what's causing it. Did you look at the heap dump? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slo w-and-can-t-update-cache-tp9676p9685.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
答复: Cache.invoke are terribly slow and can't update cache
Thanks Val! Can you explain more detail how "invoke" works internally comparing with put operation. For we pass a "function" to ignite. I see below Java Doc about "invoke" * An instance of entry processor must be stateless as it may be invoked multiple times on primary and * backup nodes in the cache. It is guaranteed that the value passed to the entry processor will be always * the same. My question: can we use *outer* object in the function process of EntryProcessor which is visible in its scope? More complex, how to iterate, just like in my code? Thanks Shawn -邮件原件- 发件人: vkulichenko [mailto:valentin.kuliche...@gmail.com] 发送时间: 2016年12月22日 2:52 收件人: user@ignite.apache.org 主题: Re: Cache.invoke are terribly slow and can't update cache Hi Shawn, Cache is not updated because you never update it :) i.e. entry.setValue() is never called. Bad performance can be caused by the fact that you do multiple operations one after another in a single thread without any batching. Consider using invokeAll or IgniteDataStreamer. As for OOME, it's hard to tell what's causing it. Did you look at the heap dump? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slo w-and-can-t-update-cache-tp9676p9685.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Cache.invoke are terribly slow and can't update cache
Hi Shawn, Cache is not updated because you never update it :) i.e. entry.setValue() is never called. Bad performance can be caused by the fact that you do multiple operations one after another in a single thread without any batching. Consider using invokeAll or IgniteDataStreamer. As for OOME, it's hard to tell what's causing it. Did you look at the heap dump? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Cache-invoke-are-terribly-slow-and-can-t-update-cache-tp9676p9685.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Cache.invoke are terribly slow and can't update cache
Hi experts,
I try to update cache by delta by calling invoke. This is the code, it is
terribly slow and run out of Memory and no cache updated. Please help.
table.getRows().forEach(r -> cache.invoke(r.getKey(), (entry, args) ->
{
BinaryObject bo = entry.getValue();
if (bo == null)
{
BinaryObjectBuilder builder =
IgniteManager.getIgnite().binary().builder(cacheName);
builder.setField("site", r.getSite());
...
bo = builder.build();
}
BinaryObjectBuilder builder = bo.toBuilder();
for (String measName : table.getSchema().getMeasNames())
{
String measValue = r.getMeasValue(table.getSchema(),
measName);
if (measValue != null)
{
builder.setField(measName, measValue);
}
}
return null;
}));
Thanks
Shawn
