Re: IgniteEvents for MVCC caches
Igniters, Igniters, I see three distinct problems with cache events, if we are to put SQL aside. 1) Strange "ENTRY_CREATED" and "ENTRY_DESTROYED" events They simply look outdated. I propose to deprecate them in the next AI release with optional flag to re-enalbe them if someone really used them. 2) Inconsistency for PUT/REMOVE/READ events Currently for different transaction types these events are fired at different times. For TRANSACTIONAL cache we fire them when entry is really read or updated, what depends on transaction mode. E.g. for PESSIMISTIC/READ_COMMITTED mode two subsequent get() will raise two events. For PESSIMISTIC/REPEATABLE_READ it will be one event. The same goes for puts. For MVCC caches it will work differently - every update will trigger separate PUT event if executed through SQL request. But not every update through PUT API will trigger an event. I think we need to come to clear semantics here, which can be explained for user. For me it could sound like this: *"If entry is read inside a transaction, at least one read event is fired. If entry is updated within transaction, at least one update event is fired."* What do you think? 3) Lock/unlock events inconsistency >From what I know "LOCKED/UNLOCKED" events has no relation to IgniteCache.lock() method family. These events are fired in any cache when entry is locked for update. This is valid for all cache modes. But due to arhcitecture differences, we cannot fire UNLOCK events for MVCC caches. Because there is no explicit UNLOCK phase anymore. That is, we know when current transaction locked it. But the only way to understand that entry is unlocked is to check it in subsequent transaction. The only way to fix is to store all affected keys in-memory, what will lead to OOME for large transactions. My proposal here is not to fire "UNLOCK" event for MVCC caches. So, to summarize, my proposal is: 1) Deprecate ENTRY_CREATED/ENTRY_DESTROYED events 2) Formulate in JavaDocs semantics of PUT/REMOVE/READ events. MVCC doesn't bring any new problems here. 3) Do not fire UNLOCK events for MVCC (explain in JavaDocs) What do you think? Vladimir. On Mon, Sep 24, 2018 at 1:28 PM Павлухин Иван wrote: > Hi guys, > > Andrey, thanks for pointing out explicit lock/unlock operations. Actually, > I think that lock/unlock events are perfectly valid for TRANSACTIONAL > caches, so there is a little chance that they could be deprecated. On the > other hand mvcc lock/unlock is not related to non-mvcc lock/unlock because > we do not support transactions involving caches with different modes. I > could imagine that lock/unlock events can be used for debugging purposes. > Currently events documentation is quite humble, e.g. for > EVT_CACHE_OBJECT_LOCKED event we have javadoc "Built-in event type: object > locked.". What object? What does "locked" mean? And "locking" semantics is > not slightly different for mvcc and non-mvcc caches (already mentioned > unlock on commit). So, another idea which came to my mind is introducing > additional event type, say MVCC_LOCKED. > > Denis, there are 2 groups of cache events: EVT_CACHE_ENTRY_<...> and > EVT_CACHE_OBJECT_<...>. EVT_CACHE_ENTRY_CREATED and > EVT_CACHE_ENTRY_DESTROYED are related to on-heap part of cache behavior. > Correct me if I am wrong, but now all caches use off-heap storage. And > EVT_CACHE_ENTRY_CREATED, EVT_CACHE_ENTRY_DESTROYED are triggered when > GridCacheMapEntry object is created as part of an access chain to values > stored in the cache. E.g. cache.put invocation causes following sequence of > events: CACHE_ENTRY_CREATED, CACHE_OBJECT_PUT, CACHE_ENTRY_DESTROYED. > Actually, there are no special problems with supporting CACHE_ENTRY_<...> > events for MVCC. And yes, I do not see much user value in them. > > Also, it seems that I chose a little bit confusing name for the thread. I > would like to keep current discussion mainly about key-value API events for > MVCC caches. SQL and events is a little bit different topic with number of > additional problems. > > 2018-09-22 17:35 GMT+03:00 Denis Magda : > > > Vladimir, > > > > That's a good question. I confused _ENTRY_ events with those mentioned by > > you. > > > > Anyway, are you saying that the _ENTRY_ events are workable but have no > > value or they are not workable any longer with the MVCC? > > > > -- > > Denis > > > > > > > > On Sat, Sep 22, 2018 at 4:10 AM Vladimir Ozerov > > wrote: > > > > > Denis, > > > > > > What is the point of these events, provided that we already has > CACHE_PUT > > > and CACHE_REMOVED? > > > > > > On Sat, Sep 22, 2018 at 1:01 AM Denis Magda wrote: > > > > > > > Hello Ivan, > > > > > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > > > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and > > > have a > > > > > little value nowadays. Does anyone have something against > deprecating > > > > them? > > > > > > > > > > > > These events are used in a pure caching or in-memory data grid use >
Re: IgniteEvents for MVCC caches
Hi guys, Andrey, thanks for pointing out explicit lock/unlock operations. Actually, I think that lock/unlock events are perfectly valid for TRANSACTIONAL caches, so there is a little chance that they could be deprecated. On the other hand mvcc lock/unlock is not related to non-mvcc lock/unlock because we do not support transactions involving caches with different modes. I could imagine that lock/unlock events can be used for debugging purposes. Currently events documentation is quite humble, e.g. for EVT_CACHE_OBJECT_LOCKED event we have javadoc "Built-in event type: object locked.". What object? What does "locked" mean? And "locking" semantics is not slightly different for mvcc and non-mvcc caches (already mentioned unlock on commit). So, another idea which came to my mind is introducing additional event type, say MVCC_LOCKED. Denis, there are 2 groups of cache events: EVT_CACHE_ENTRY_<...> and EVT_CACHE_OBJECT_<...>. EVT_CACHE_ENTRY_CREATED and EVT_CACHE_ENTRY_DESTROYED are related to on-heap part of cache behavior. Correct me if I am wrong, but now all caches use off-heap storage. And EVT_CACHE_ENTRY_CREATED, EVT_CACHE_ENTRY_DESTROYED are triggered when GridCacheMapEntry object is created as part of an access chain to values stored in the cache. E.g. cache.put invocation causes following sequence of events: CACHE_ENTRY_CREATED, CACHE_OBJECT_PUT, CACHE_ENTRY_DESTROYED. Actually, there are no special problems with supporting CACHE_ENTRY_<...> events for MVCC. And yes, I do not see much user value in them. Also, it seems that I chose a little bit confusing name for the thread. I would like to keep current discussion mainly about key-value API events for MVCC caches. SQL and events is a little bit different topic with number of additional problems. 2018-09-22 17:35 GMT+03:00 Denis Magda : > Vladimir, > > That's a good question. I confused _ENTRY_ events with those mentioned by > you. > > Anyway, are you saying that the _ENTRY_ events are workable but have no > value or they are not workable any longer with the MVCC? > > -- > Denis > > > > On Sat, Sep 22, 2018 at 4:10 AM Vladimir Ozerov > wrote: > > > Denis, > > > > What is the point of these events, provided that we already has CACHE_PUT > > and CACHE_REMOVED? > > > > On Sat, Sep 22, 2018 at 1:01 AM Denis Magda wrote: > > > > > Hello Ivan, > > > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and > > have a > > > > little value nowadays. Does anyone have something against deprecating > > > them? > > > > > > > > > These events are used in a pure caching or in-memory data grid use > cases > > > where key-value is a primary access pattern. I personally know several > > > companies who rely on those events. So, we can't discontinue them, not > > > everyone uses SQL. What's a hurdle of making them working with MVCC? > > > > > > -- > > > Denis > > > > > > On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван > > wrote: > > > > > > > Hi Igniters, > > > > > > > > As you might know MVCC was introduced in Apache Ignite. We started > > > > IgniteEvents support implementation for MVCC caches and faced some > > > > obstacles. I would like to start a discussion about next steps which > > > should > > > > be done to deal with current problems. The main theme is about > > > > EventType.EVTS_CACHE and key-value API. > > > > > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and > > have a > > > > little value nowadays. Does anyone have something against deprecating > > > them? > > > > > > > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, > > > > because for MVCC all locks are released on transaction end. And it > does > > > not > > > > sound good idea to track all locked entries during transaction > > execution, > > > > in MVCC we are not keeping entries modified participating in > > transaction > > > in > > > > private working set in memory. One possible solution is deprecation > of > > > > lock, unlock events. Another one is introducing special lock event > for > > > > MVCC, but it will be confusing. Also, I see an option of firing only > > > > EVT_CACHE_OBJECT_LOCKED. > > > > > > > > Last, is different number of events fired for similar scenarios and > > > > different cache modes. Let's consider "put, remove, put" for the same > > key > > > > and different modes. For ATOMIC 2 put and 1 remove event will be > fired. > > > For > > > > TRANSACTIONAL 1 put will be fired in case of commit and nothing in > case > > > of > > > > rollback. For MVCC in current vision 2 put will be fired regardless > > > whether > > > > transaction was committed and rolled back. Currently I do not see > > options > > > > how to overcome it. > > > > > > > > Also, I hardly imagine current use cases for cache events. I think > that > > > > understanding them is the best way for developing working solution >
Re: IgniteEvents for MVCC caches
Vladimir, That's a good question. I confused _ENTRY_ events with those mentioned by you. Anyway, are you saying that the _ENTRY_ events are workable but have no value or they are not workable any longer with the MVCC? -- Denis On Sat, Sep 22, 2018 at 4:10 AM Vladimir Ozerov wrote: > Denis, > > What is the point of these events, provided that we already has CACHE_PUT > and CACHE_REMOVED? > > On Sat, Sep 22, 2018 at 1:01 AM Denis Magda wrote: > > > Hello Ivan, > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and > have a > > > little value nowadays. Does anyone have something against deprecating > > them? > > > > > > These events are used in a pure caching or in-memory data grid use cases > > where key-value is a primary access pattern. I personally know several > > companies who rely on those events. So, we can't discontinue them, not > > everyone uses SQL. What's a hurdle of making them working with MVCC? > > > > -- > > Denis > > > > On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван > wrote: > > > > > Hi Igniters, > > > > > > As you might know MVCC was introduced in Apache Ignite. We started > > > IgniteEvents support implementation for MVCC caches and faced some > > > obstacles. I would like to start a discussion about next steps which > > should > > > be done to deal with current problems. The main theme is about > > > EventType.EVTS_CACHE and key-value API. > > > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and > have a > > > little value nowadays. Does anyone have something against deprecating > > them? > > > > > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, > > > because for MVCC all locks are released on transaction end. And it does > > not > > > sound good idea to track all locked entries during transaction > execution, > > > in MVCC we are not keeping entries modified participating in > transaction > > in > > > private working set in memory. One possible solution is deprecation of > > > lock, unlock events. Another one is introducing special lock event for > > > MVCC, but it will be confusing. Also, I see an option of firing only > > > EVT_CACHE_OBJECT_LOCKED. > > > > > > Last, is different number of events fired for similar scenarios and > > > different cache modes. Let's consider "put, remove, put" for the same > key > > > and different modes. For ATOMIC 2 put and 1 remove event will be fired. > > For > > > TRANSACTIONAL 1 put will be fired in case of commit and nothing in case > > of > > > rollback. For MVCC in current vision 2 put will be fired regardless > > whether > > > transaction was committed and rolled back. Currently I do not see > options > > > how to overcome it. > > > > > > Also, I hardly imagine current use cases for cache events. I think that > > > understanding them is the best way for developing working solution for > > > MVCC. > > > > > > I need your opinions. > > > > > > 2018-09-21 12:54 GMT+03:00 Павлухин Иван : > > > > > > > Hi Igniters, > > > > > > > > As you might know MVCC was introduced in Apache Ignite. We started > > > > IgniteEvents implementation for MVCC caches and faced some > obstacles. I > > > > would like to start a discussion about next steps which should be > done > > to > > > > deal with current problems. > > > > > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > >
Re: IgniteEvents for MVCC caches
Denis, What is the point of these events, provided that we already has CACHE_PUT and CACHE_REMOVED? On Sat, Sep 22, 2018 at 1:01 AM Denis Magda wrote: > Hello Ivan, > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a > > little value nowadays. Does anyone have something against deprecating > them? > > > These events are used in a pure caching or in-memory data grid use cases > where key-value is a primary access pattern. I personally know several > companies who rely on those events. So, we can't discontinue them, not > everyone uses SQL. What's a hurdle of making them working with MVCC? > > -- > Denis > > On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван wrote: > > > Hi Igniters, > > > > As you might know MVCC was introduced in Apache Ignite. We started > > IgniteEvents support implementation for MVCC caches and faced some > > obstacles. I would like to start a discussion about next steps which > should > > be done to deal with current problems. The main theme is about > > EventType.EVTS_CACHE and key-value API. > > > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a > > little value nowadays. Does anyone have something against deprecating > them? > > > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, > > because for MVCC all locks are released on transaction end. And it does > not > > sound good idea to track all locked entries during transaction execution, > > in MVCC we are not keeping entries modified participating in transaction > in > > private working set in memory. One possible solution is deprecation of > > lock, unlock events. Another one is introducing special lock event for > > MVCC, but it will be confusing. Also, I see an option of firing only > > EVT_CACHE_OBJECT_LOCKED. > > > > Last, is different number of events fired for similar scenarios and > > different cache modes. Let's consider "put, remove, put" for the same key > > and different modes. For ATOMIC 2 put and 1 remove event will be fired. > For > > TRANSACTIONAL 1 put will be fired in case of commit and nothing in case > of > > rollback. For MVCC in current vision 2 put will be fired regardless > whether > > transaction was committed and rolled back. Currently I do not see options > > how to overcome it. > > > > Also, I hardly imagine current use cases for cache events. I think that > > understanding them is the best way for developing working solution for > > MVCC. > > > > I need your opinions. > > > > 2018-09-21 12:54 GMT+03:00 Павлухин Иван : > > > > > Hi Igniters, > > > > > > As you might know MVCC was introduced in Apache Ignite. We started > > > IgniteEvents implementation for MVCC caches and faced some obstacles. I > > > would like to start a discussion about next steps which should be done > to > > > deal with current problems. > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > >
Re: IgniteEvents for MVCC caches
Hello Ivan, First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a > little value nowadays. Does anyone have something against deprecating them? These events are used in a pure caching or in-memory data grid use cases where key-value is a primary access pattern. I personally know several companies who rely on those events. So, we can't discontinue them, not everyone uses SQL. What's a hurdle of making them working with MVCC? -- Denis On Fri, Sep 21, 2018 at 6:44 AM Павлухин Иван wrote: > Hi Igniters, > > As you might know MVCC was introduced in Apache Ignite. We started > IgniteEvents support implementation for MVCC caches and faced some > obstacles. I would like to start a discussion about next steps which should > be done to deal with current problems. The main theme is about > EventType.EVTS_CACHE and key-value API. > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a > little value nowadays. Does anyone have something against deprecating them? > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, > because for MVCC all locks are released on transaction end. And it does not > sound good idea to track all locked entries during transaction execution, > in MVCC we are not keeping entries modified participating in transaction in > private working set in memory. One possible solution is deprecation of > lock, unlock events. Another one is introducing special lock event for > MVCC, but it will be confusing. Also, I see an option of firing only > EVT_CACHE_OBJECT_LOCKED. > > Last, is different number of events fired for similar scenarios and > different cache modes. Let's consider "put, remove, put" for the same key > and different modes. For ATOMIC 2 put and 1 remove event will be fired. For > TRANSACTIONAL 1 put will be fired in case of commit and nothing in case of > rollback. For MVCC in current vision 2 put will be fired regardless whether > transaction was committed and rolled back. Currently I do not see options > how to overcome it. > > Also, I hardly imagine current use cases for cache events. I think that > understanding them is the best way for developing working solution for > MVCC. > > I need your opinions. > > 2018-09-21 12:54 GMT+03:00 Павлухин Иван : > > > Hi Igniters, > > > > As you might know MVCC was introduced in Apache Ignite. We started > > IgniteEvents implementation for MVCC caches and faced some obstacles. I > > would like to start a discussion about next steps which should be done to > > deal with current problems. > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > > > -- > Best regards, > Ivan Pavlukhin >
Re: IgniteEvents for MVCC caches
Hi Ivan, Lock\Unlock events can be used for explicit lock once IGNITE-9324 will be resolved. Most likely we should rethink Lock\Unlock events semantic for mvcc mode rather then just deprecate them. For update events we can provide tx XID as well to allow entry events and TX commit\rollback events being tracked on user side So, we can rely events consistency issue on user. [1] https://issues.apache.org/jira/browse/IGNITE-9324 On Fri, Sep 21, 2018 at 1:44 PM Павлухин Иван wrote: > Hi Igniters, > > As you might know MVCC was introduced in Apache Ignite. We started > IgniteEvents support implementation for MVCC caches and faced some > obstacles. I would like to start a discussion about next steps which should > be done to deal with current problems. The main theme is about > EventType.EVTS_CACHE and key-value API. > > First of all, it looks like that EVT_CACHE_ENTRY_CREATED, > EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a > little value nowadays. Does anyone have something against deprecating them? > > Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, > because for MVCC all locks are released on transaction end. And it does not > sound good idea to track all locked entries during transaction execution, > in MVCC we are not keeping entries modified participating in transaction in > private working set in memory. One possible solution is deprecation of > lock, unlock events. Another one is introducing special lock event for > MVCC, but it will be confusing. Also, I see an option of firing only > EVT_CACHE_OBJECT_LOCKED. > > Last, is different number of events fired for similar scenarios and > different cache modes. Let's consider "put, remove, put" for the same key > and different modes. For ATOMIC 2 put and 1 remove event will be fired. For > TRANSACTIONAL 1 put will be fired in case of commit and nothing in case of > rollback. For MVCC in current vision 2 put will be fired regardless whether > transaction was committed and rolled back. Currently I do not see options > how to overcome it. > > Also, I hardly imagine current use cases for cache events. I think that > understanding them is the best way for developing working solution for > MVCC. > > I need your opinions. > > 2018-09-21 12:54 GMT+03:00 Павлухин Иван : > > > Hi Igniters, > > > > As you might know MVCC was introduced in Apache Ignite. We started > > IgniteEvents implementation for MVCC caches and faced some obstacles. I > > would like to start a discussion about next steps which should be done to > > deal with current problems. > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > > > -- > Best regards, > Ivan Pavlukhin > -- Best regards, Andrey V. Mashenkov
Re: IgniteEvents for MVCC caches
Hi Igniters, As you might know MVCC was introduced in Apache Ignite. We started IgniteEvents support implementation for MVCC caches and faced some obstacles. I would like to start a discussion about next steps which should be done to deal with current problems. The main theme is about EventType.EVTS_CACHE and key-value API. First of all, it looks like that EVT_CACHE_ENTRY_CREATED, EVT_CACHE_ENTRY_DESTROYED are legacy came from Ignite 1.x time and have a little value nowadays. Does anyone have something against deprecating them? Second, there could be problems with EVT_CACHE_OBJECT_UNLOCKED event, because for MVCC all locks are released on transaction end. And it does not sound good idea to track all locked entries during transaction execution, in MVCC we are not keeping entries modified participating in transaction in private working set in memory. One possible solution is deprecation of lock, unlock events. Another one is introducing special lock event for MVCC, but it will be confusing. Also, I see an option of firing only EVT_CACHE_OBJECT_LOCKED. Last, is different number of events fired for similar scenarios and different cache modes. Let's consider "put, remove, put" for the same key and different modes. For ATOMIC 2 put and 1 remove event will be fired. For TRANSACTIONAL 1 put will be fired in case of commit and nothing in case of rollback. For MVCC in current vision 2 put will be fired regardless whether transaction was committed and rolled back. Currently I do not see options how to overcome it. Also, I hardly imagine current use cases for cache events. I think that understanding them is the best way for developing working solution for MVCC. I need your opinions. 2018-09-21 12:54 GMT+03:00 Павлухин Иван : > Hi Igniters, > > As you might know MVCC was introduced in Apache Ignite. We started > IgniteEvents implementation for MVCC caches and faced some obstacles. I > would like to start a discussion about next steps which should be done to > deal with current problems. > > -- > Best regards, > Ivan Pavlukhin > -- Best regards, Ivan Pavlukhin
IgniteEvents for MVCC caches
Hi Igniters, As you might know MVCC was introduced in Apache Ignite. We started IgniteEvents implementation for MVCC caches and faced some obstacles. I would like to start a discussion about next steps which should be done to deal with current problems. -- Best regards, Ivan Pavlukhin
