Hi,
It is time to continue DataStreamer for MVCC caches discussion. Main
focus is on allowOverwrite=false mode. Currently there is a problem
related to partition update counters.
BACKGROUND
As you might know MVCC transactions update partition counters on
transaction finish phase and on backups
On Tue, Aug 14, 2018 at 4:30 AM, Vladimir Ozerov
wrote:
> Bypassing WAL will make the whole cache data vulnerable to complete loss in
> case of node failure. I would not do this automatically.
>
Well, in this case I would expect a log message suggesting that there is an
option to turn off WAL
Bypassing WAL will make the whole cache data vulnerable to complete loss in
case of node failure. I would not do this automatically.
On Mon, Jul 16, 2018 at 12:28 PM Ilya Kasnacheev
wrote:
> Hello!
>
> Can we also bypass WAL for such mode automatically?
>
> However, we will definitely need a
Hello!
Can we also bypass WAL for such mode automatically?
However, we will definitely need a 'normal' mode of DataStreamer operation,
for people who use dataStreamer with custom stream transformers on existing
data in use.
Regards,
--
Ilya Kasnacheev
2018-07-14 12:33 GMT+03:00 Vladimir
Igniters,
Denis is right - please pay attention to IEP-22, as this is how we are
going to load data into the grid in future. Note that current data streamer
internals are not efficient enough, primarily because it has to interact
with page memory, free lists and various BTree's in regular manner.
Agree that initial loading and real-time streaming should be seen as
different use cases.
For the loading part, I would borrow ideas from direct data load IEP [1].
Ignite should assume that no app works with the cluster until it's
preloaded. So, no global locks or things like that. Just fasten a
Ivan,
Anyway DataStreamer is the fastest way to deliver data to a data node, the
question is how to apply it correctly.
I don’t thing we need one more tool, which 90% is the same as DataStreamer.
All we need is just to implement a couple of new stream receivers.
Regards,
Igor
> 13 июля 2018
Hi Igniters,
I had a look into IgniteDataStreamer. As far as I understand, currently it
just works incorrectly for MVCC tables. It appears as a blocker for
releasing MVCC. The simplest thing is to refuse creating streamer for MVCC
tables.
Next step could be hair splitting of related use cases.
Yakov,
We can introduce several modes:
1) initial loading which replaces data (allowOverwrite=true) with initial
version or leaves it as is (allowOverwrite=false) and requires exclusive table
lock (fastest one)
2) continuous loading which has its own version and links the data as regular
Igor,
I can't say if I agree with any of the suggestions. I would like us to
start from answering the question - what is data streamer used for?
First of all, for initial data loading. This should be super fast mode
probably ignoring all transactional semantics, but providing certain
guarantees
Hi Igniters,
You know, we're currently developing fair MVCC for transactional caches (
https://issues-test.apache.org/jira/plugins/servlet/mobile#issue/IGNITE-4191
).
At now we're trying to make work IgniteDataStreamer with MVCC.
The problem is that MVCC requires linking between versions
11 matches
Mail list logo
