Hello

[Дмитрий Рябов]

Hello, I want to work with you. Can you give me access to contribute? My
username in JIRA is SomeFire.

IGNITE-602

[Дмитрий Рябов]

Hello, can you answer me a simple question?

We have 2 looping lists:

ArrayList list1 = new ArrayList<>();
ArrayList list2 = new ArrayList<>();

list2.add(list1);
list1.add(list2);

Class GridToStringBuilder represent them into string.

Simple public static  String toString(Class cls, T obj) method
returns string "ArrayList [size=1]" for each list.

*Question is* how should look the result of toString(Class cls, T obj,
String name, @Nullable Object val) method and other methods with additional
values?
"ArrayList [size=1, name=ArrayList[size=1]]" or something else?

The trouble is in GridStringBuilder.a(Object obj) method - it tries to
append collection's toString() method which invokes toString() on every
element.

Re: IGNITE-602

[Дмитрий Рябов]

Hello, Vladimir. Do you mean something like "ArrayList [size=1,
name=ArrayList [size=1, ArrayList {position 0}]]"?

2017-02-16 12:32 GMT+03:00 Vladimir Ozerov :

> Hi Dmitriy,
>
> We should print all objects, because this is the whole purpose of
> GridToStringBuilder class - to print as much as possible in pretty format.
> For cyclic references, we should introduce smarter approach, ensuring that
> certain object is printed only once, I would use a kind of stack or
> IdentityHashMap to track already printed entries and then print a reference
> to them instead of the whole body.
>
> We use similar approach for binary objects, see
> BinaryObjectExImpl.toString(...)
> method. In this case (hash + body) is printed normally, but when a cycle is
> found, we print only (hash). This is not very suitable for
> GridToStringBuilder, as we definitely do not want output to be flooded with
> hashes, which are useless in common case. May be it makes sense to object's
> position in already printed string.
>
> Vladimir.
>
>
> On Wed, Feb 15, 2017 at 3:07 PM, Дмитрий Рябов 
> wrote:
>
> > Hello, can you answer me a simple question?
> >
> > We have 2 looping lists:
> >
> > ArrayList list1 = new ArrayList<>();
> > ArrayList list2 = new ArrayList<>();
> >
> > list2.add(list1);
> > list1.add(list2);
> >
> > Class GridToStringBuilder represent them into string.
> >
> > Simple public static  String toString(Class cls, T obj) method
> > returns string "ArrayList [size=1]" for each list.
> >
> > *Question is* how should look the result of toString(Class cls, T obj,
> > String name, @Nullable Object val) method and other methods with
> additional
> > values?
> > "ArrayList [size=1, name=ArrayList[size=1]]" or something else?
> >
> > The trouble is in GridStringBuilder.a(Object obj) method - it tries to
> > append collection's toString() method which invokes toString() on every
> > element.
> >
>

IGNITE-4404 long running test refactoring - how to refactor?

[Дмитрий Рябов]

Hello, community. Ticket description:

in testTransform

final int THREADS = 5;
final int ITERATIONS_PER_THREAD = 10_000;

So, what should I change?
Other methods have

final int THREADS = 5;
final int ITERATIONS_PER_THREAD = iterations();

where iterations() {return 10_000;}

Should I use iterations() and create same method for threads? Or may be
replace all local finals to class finals?

Should I do something else? Couse the only way to make test shorter is to
reduce number of iterations, but purpose of this test is long concurrent
actions.

IGNITE-602 - ready for review

[Дмитрий Рябов]

Hello. Please, review.

https://issues.apache.org/jira/browse/IGNITE-602 [Test] GridToStringBuilder
is vulnerable for StackOverflowError caused by infinite recursion

CI tests:
http://ci.ignite.apache.org/project.html?projectId=IgniteTes
ts&branch_IgniteTests=pull%2F1558%2Fhead

Re: IGNITE-4404 long running test refactoring - how to refactor?

[Дмитрий Рябов]

I think we can keep this numbers for atomic behavior and decrease for
transactional.

But which tests can be replaced with mocks if they all use node's cache?

2017-02-22 12:47 GMT+03:00 Константин Дудков :

> Answered to ticket. It's more about testing process than real refactoring
> so I think we need to discuss it further.
>
> 21.02.2017, 17:54, "Yakov Zhdanov" :
> > Konstantin Dudkov, you created ticket some time ago. Can you please
> clarify?
> >
> > --Yakov
>
> --
> С уважением,
> Константин Дудков
> kdud...@ya.ru
>

IGNITE-1393 futures crash ignite when trying to get lock for stopped cache

[Дмитрий Рябов]

The problem appears when future tries to finish itself and it trying to get
lock when cache is already stopped. The only way to solve this issue is
catching such assertion errors. At least, we are already catching some
exceptions in the test.

But what about real situation? Should I add catch assert in the test or add
cache check in transaction/future?

java.lang.AssertionError: Lock is not owned for commit...
Suppressed: java.lang.IllegalStateException: Grid is in invalid state to
perform this operation. It either not started yet or has already being or
have stopped [gridName=cache.GridCacheStopSelfTest0, state=STOPPED]

Re: [jira] [Created] (IGNITE-4794) Reconsider CachePeekMode.ONHEAP

[Дмитрий Рябов]

Onheap will be changed? Where I can read plans about future versions? Is
there anything except tickets?

2017-03-06 23:31 GMT+03:00 Denis Magda (JIRA) :

> Denis Magda created IGNITE-4794:
> ---
>
>  Summary: Reconsider CachePeekMode.ONHEAP
>  Key: IGNITE-4794
>  URL: https://issues.apache.org/jira/browse/IGNITE-4794
>  Project: Ignite
>   Issue Type: Sub-task
> Reporter: Denis Magda
>
>
> In Apache Ignite 2.0 we will no longer have {{ONHEAP}} tier in a sense it
> was supported for Apache Ignite 1.x releases. This is why we need to
> revisit {{CachePeekMode}} interface and adjust an implementation based on
> {{CachePeekMode.ONHEAP}}.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)
>

IGNITE-1393 - ready for review

[Дмитрий Рябов]

Hello. Please, review.

https://issues.apache.org/jira/browse/IGNITE-1393 [Test] AssertionError
when stop node executing transaction - GridCacheStopSelfTest

Re: distributed transaction of non-single coordinator

[Дмитрий Рябов]

Alexey Goncharuk, heh, my initial understanding was that transferring of tx
ownership from one node to another will be happened automatically when
originating node is gone down.

2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV :

> Im aiming to span transaction on multiple threads, nodes, jvms(soon). So
> every node is able to rollback, or commit common transaction.It turned up i
> need to transfer tx between nodes in order to commit transaction in
> different node(in the same jvm).
>
> пт, 10 мар. 2017 г. в 15:20, Alexey Goncharuk  >:
>
> > Aleksey,
> >
> > Do you mean that you want a concept of transferring of tx ownership from
> > one node to another? My initial understanding was that you want to be
> able
> > to update keys in a transaction from multiple threads in parallel.
> >
> > --AG
> >
> > 2017-03-10 15:01 GMT+03:00 ALEKSEY KUZNETSOV :
> >
> > > Well. Consider transaction started in one node, and continued in
> another
> > > one.
> > > The following test describes my idea:
> > >
> > > Ignite ignite1 = ignite(0);
> > >
> > > IgniteTransactions transactions = ignite1.transactions();
> > >
> > > IgniteCache cache = ignite1.getOrCreateCache("
> > > testCache");
> > >
> > > Transaction tx = transactions.txStart(concurrency, isolation);
> > >
> > > cache.put("key1", 1);
> > >
> > > cache.put("key2", 2);
> > >
> > > tx.stop();
> > >
> > > IgniteInternalFuture fut = GridTestUtils.runAsync(() -> {
> > > IgniteTransactions ts = ignite(1).transactions();
> > > Assert.assertNull(ts.tx());
> > > Assert.assertEquals(TransactionState.STOPPED, tx.state());
> > > ts.txStart(tx);
> > > Assert.assertEquals(TransactionState.ACTIVE, tx.state());
> > > cache.put("key3", 3);
> > > Assert.assertTrue(cache.remove("key2"));
> > > tx.commit();
> > > return true;
> > > });
> > >
> > > fut.get();
> > >
> > > Assert.assertEquals(TransactionState.COMMITTED, tx.state());
> > > Assert.assertEquals((long)1, (long)cache.get("key1"));
> > > Assert.assertEquals((long)3, (long)cache.get("key3"));
> > > Assert.assertFalse(cache.containsKey("key2"));
> > >
> > > In method *ts.txStart(...)* we just rebind *tx* to current thread:
> > >
> > > public void txStart(Transaction tx) {
> > > TransactionProxyImpl transactionProxy = (TransactionProxyImpl)tx;
> > > cctx.tm().reopenTx(transactionProxy.tx());
> > > transactionProxy.bindToCurrentThread();
> > > }
> > >
> > > In method *reopenTx* we alter *threadMap* so that it binds transaction
> > > to current thread.
> > >
> > > How do u think about it ?
> > >
> > >
> > > вт, 7 мар. 2017 г. в 22:38, Denis Magda :
> > >
> > > > Hi Alexey,
> > > >
> > > > Please share the rational behind this and the thoughts, design ideas
> > you
> > > > have in mind.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY KUZNETSOV <
> > > alkuznetsov...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi all! Im designing distributed transaction which can be started
> at
> > > one
> > > > > node, and continued at other one. Has anybody thoughts on it ?
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > >
> > > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

Re: distributed transaction of non-single coordinator

[Дмитрий Рябов]

What about to send special message to random/choosen node when we start
transaction? And when rollback procedure begins - this second node will
check state of originating node and if it is down then second node became
originating?

2017-03-10 17:25 GMT+03:00 Alexey Goncharuk :

> Aleksey,
>
> I think I am starting to get what you want, but I have a few concerns:
>  - What is the API for the proposed change? In your test, you pass an
> instance of transaction created on ignite(0) to the ignite instance
> ignite(1). This is obviously not possible in a truly distributed
> (multi-jvm) environment.
> - How will you synchronize cache update actions and transaction commit?
> Say, you have one node that decided to commit, but another node is still
> writing within this transaction. How do you make sure that two nodes will
> not call commit() and rollback() simultaneously?
>  - How do you make sure that either commit() or rollback() is called if an
> originator failed?
>
> 2017-03-10 15:38 GMT+03:00 Дмитрий Рябов :
>
> > Alexey Goncharuk, heh, my initial understanding was that transferring of
> tx
> > ownership from one node to another will be happened automatically when
> > originating node is gone down.
> >
> > 2017-03-10 15:36 GMT+03:00 ALEKSEY KUZNETSOV :
> >
> > > Im aiming to span transaction on multiple threads, nodes, jvms(soon).
> So
> > > every node is able to rollback, or commit common transaction.It turned
> > up i
> > > need to transfer tx between nodes in order to commit transaction in
> > > different node(in the same jvm).
> > >
> > > пт, 10 мар. 2017 г. в 15:20, Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > >:
> > >
> > > > Aleksey,
> > > >
> > > > Do you mean that you want a concept of transferring of tx ownership
> > from
> > > > one node to another? My initial understanding was that you want to be
> > > able
> > > > to update keys in a transaction from multiple threads in parallel.
> > > >
> > > > --AG
> > > >
> > > > 2017-03-10 15:01 GMT+03:00 ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com
> > >:
> > > >
> > > > > Well. Consider transaction started in one node, and continued in
> > > another
> > > > > one.
> > > > > The following test describes my idea:
> > > > >
> > > > > Ignite ignite1 = ignite(0);
> > > > >
> > > > > IgniteTransactions transactions = ignite1.transactions();
> > > > >
> > > > > IgniteCache cache = ignite1.getOrCreateCache("
> > > > > testCache");
> > > > >
> > > > > Transaction tx = transactions.txStart(concurrency, isolation);
> > > > >
> > > > > cache.put("key1", 1);
> > > > >
> > > > > cache.put("key2", 2);
> > > > >
> > > > > tx.stop();
> > > > >
> > > > > IgniteInternalFuture fut = GridTestUtils.runAsync(() -> {
> > > > > IgniteTransactions ts = ignite(1).transactions();
> > > > > Assert.assertNull(ts.tx());
> > > > > Assert.assertEquals(TransactionState.STOPPED, tx.state());
> > > > > ts.txStart(tx);
> > > > > Assert.assertEquals(TransactionState.ACTIVE, tx.state());
> > > > > cache.put("key3", 3);
> > > > > Assert.assertTrue(cache.remove("key2"));
> > > > > tx.commit();
> > > > > return true;
> > > > > });
> > > > >
> > > > > fut.get();
> > > > >
> > > > > Assert.assertEquals(TransactionState.COMMITTED, tx.state());
> > > > > Assert.assertEquals((long)1, (long)cache.get("key1"));
> > > > > Assert.assertEquals((long)3, (long)cache.get("key3"));
> > > > > Assert.assertFalse(cache.containsKey("key2"));
> > > > >
> > > > > In method *ts.txStart(...)* we just rebind *tx* to current thread:
> > > > >
> > > > > public void txStart(Transaction tx) {
> > > > > TransactionProxyImpl transactionProxy =
> (TransactionProxyImpl)tx;
> > > > > cctx.tm().reopenTx(transactionProxy.tx());
> > > > > transactionProxy.bindToCurrentThread();
> > > > > }
> > > > >
> > > > > In method *reopenTx* we alter *threadMap* so that it binds
> > transaction
> > > > > to current thread.
> > > > >
> > > > > How do u think about it ?
> > > > >
> > > > >
> > > > > вт, 7 мар. 2017 г. в 22:38, Denis Magda :
> > > > >
> > > > > > Hi Alexey,
> > > > > >
> > > > > > Please share the rational behind this and the thoughts, design
> > ideas
> > > > you
> > > > > > have in mind.
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Mar 7, 2017, at 3:19 AM, ALEKSEY KUZNETSOV <
> > > > > alkuznetsov...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi all! Im designing distributed transaction which can be
> started
> > > at
> > > > > one
> > > > > > > node, and continued at other one. Has anybody thoughts on it ?
> > > > > > > --
> > > > > > >
> > > > > > > *Best Regards,*
> > > > > > >
> > > > > > > *Kuznetsov Aleksey*
> > > > > >
> > > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > > >
> > > >
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
>

Ignite-4795 - ready for review

[Дмитрий Рябов]

Hello, community. Please, review and/or suggest something about javadocs of
transactions.

PR: https://github.com/apache/ignite/pull/1630/files

JIRA: https://issues.apache.org/jira/browse/IGNITE-4795

Re: Ignite-4795 - ready for review

[Дмитрий Рябов]

Done.

2017-03-20 16:30 GMT+03:00 Антон Чураев :

> Dmitry, thank you!
>
> Could you please also change issue status to "Patch available".
>
> 2017-03-20 16:01 GMT+03:00 Дмитрий Рябов :
>
> > Hello, community. Please, review and/or suggest something about javadocs
> of
> > transactions.
> >
> > PR: https://github.com/apache/ignite/pull/1630/files
> >
> > JIRA: https://issues.apache.org/jira/browse/IGNITE-4795
> >
>
>
>
> --
> С уважением,
> Чураев Антон
>

Re: strange GridCacheReturn

[Дмитрий Рябов]

Object with value and success flag. It is used in futures to understand
what is result of asyncronious action.

2017-03-23 13:26 GMT+03:00 ALEKSEY KUZNETSOV :

> Hi all! what is the use of GridCacheReturn ?
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

Re: Ignite-4795 - ready for review

[Дмитрий Рябов]

Hello, can anyone review this issue?

2017-03-20 16:33 GMT+03:00 Дмитрий Рябов :

> Done.
>
> 2017-03-20 16:30 GMT+03:00 Антон Чураев :
>
>> Dmitry, thank you!
>>
>> Could you please also change issue status to "Patch available".
>>
>> 2017-03-20 16:01 GMT+03:00 Дмитрий Рябов :
>>
>> > Hello, community. Please, review and/or suggest something about
>> javadocs of
>> > transactions.
>> >
>> > PR: https://github.com/apache/ignite/pull/1630/files
>> >
>> > JIRA: https://issues.apache.org/jira/browse/IGNITE-4795
>> >
>>
>>
>>
>> --
>> С уважением,
>> Чураев Антон
>>
>
>

Re: IgniteFuture adapters

[Дмитрий Рябов]

IgniteFuture have method

public  IgniteFuture chain(IgniteClosure, T>
doneCb);

which do this.

2017-03-27 13:30 GMT+03:00 Sergei Egorov :

> Hi!
>
> Would be nice if igniteFuture would provide a small but very usable method:
>
> public  R to(Function, R> transformer)
>
> it will allow to chain it like:
>
> compute.runAsync(runnable).to(rx()).timeout(5_000).subscribe()
>
> Where rx() is just a static function with something like:
>
>public static  Function, Observable> rx()
>
>
> WDYT?
>


2017-03-27 13:30 GMT+03:00 Sergei Egorov :

> Hi!
>
> Would be nice if igniteFuture would provide a small but very usable method:
>
> public  R to(Function, R> transformer)
>
> it will allow to chain it like:
>
> compute.runAsync(runnable).to(rx()).timeout(5_000).subscribe()
>
> Where rx() is just a static function with something like:
>
>public static  Function, Observable> rx()
>
>
> WDYT?
>

Re: IgniteFuture adapters

[Дмитрий Рябов]

Em... When you get result of your future, closure will produce new future
with completely different type.

2017-03-27 13:43 GMT+03:00 Sergei Egorov :

> It doesn't :)
>
> It returns another IgniteFuture where I want to transform it to completely
> different type.
>
> On Mon, Mar 27, 2017 at 1:41 PM Дмитрий Рябов 
> wrote:
>
> > IgniteFuture have method
> >
> > public  IgniteFuture chain(IgniteClosure,
> T>
> > doneCb);
> >
> > which do this.
> >
> > 2017-03-27 13:30 GMT+03:00 Sergei Egorov :
> >
> > > Hi!
> > >
> > > Would be nice if igniteFuture would provide a small but very usable
> > method:
> > >
> > > public  R to(Function, R> transformer)
> > >
> > > it will allow to chain it like:
> > >
> > > compute.runAsync(runnable).to(rx()).timeout(5_000).subscribe()
> > >
> > > Where rx() is just a static function with something like:
> > >
> > >public static  Function, Observable> rx()
> > >
> > >
> > > WDYT?
> > >
> >
> >
> > 2017-03-27 13:30 GMT+03:00 Sergei Egorov :
> >
> > > Hi!
> > >
> > > Would be nice if igniteFuture would provide a small but very usable
> > method:
> > >
> > > public  R to(Function, R> transformer)
> > >
> > > it will allow to chain it like:
> > >
> > > compute.runAsync(runnable).to(rx()).timeout(5_000).subscribe()
> > >
> > > Where rx() is just a static function with something like:
> > >
> > >public static  Function, Observable> rx()
> > >
> > >
> > > WDYT?
> > >
> >
>

IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

Hello, community. How should savepoint and rollback to savepoint react to
the atomic caches inside transaction? I think they should ignore such
entries, but I want to know opinion of more competent igniters.

https://issues.apache.org/jira/browse/IGNITE-4188

Re: IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

I'd prefer ignoring them instead of throwing exceptions - couse user can
use different cache configurations inside one tx and user can create some
logic depending on cached values in any cache.

2017-03-28 13:08 GMT+03:00 Yakov Zhdanov :

> As far as I know operations on atomic caches are applied immediately
> dishonoring any tx context.
>
> I would suggest that atomic cache update operation called from active tx
> throws illegal state exception, unless user intentionally permits this
> update by calling atomicCache.withAllowInTx() (similar to withSkipStore()).
>
> Thoughts?
>
> --Yakov
>

Re: IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

I mean ignore it in savepoints and rollbacks to savepoints becouse
transaction rollbacks already ignore it.

2017-03-28 18:10 GMT+03:00 Yakov Zhdanov :

> Ignore atomicCache.put(k, v) if it is called inside transaction? This is
> very counter intuitive. I am strongly against that.
>
> --Yakov
>

Re: IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

Finish savepoints or flag&exceptions for atomic operations?
Not sure about savepoints. Exceptions - yes.
https://issues.apache.org/jira/browse/IGNITE-2313 isn't it?

2017-03-29 2:12 GMT+03:00 Denis Magda :

> If we want to make the exception based approach the default one then the
> task has to be released in 2.0.
>
> Dmitriy Ryabov, do you think you can finish it (dev, review, QA) by the
> code freeze data (April 14)?
>
> —
> Denis
>
> > On Mar 28, 2017, at 11:57 AM, Dmitriy Setrakyan 
> wrote:
> >
> > On Tue, Mar 28, 2017 at 11:54 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> >> I think updating an Atomic cache from within a transaction perfectly
> makes
> >> sense. For example for some kind of operations logging and so forth.
> Still
> >> I agree that this can be error prone and forbidden by default. I agree
> with
> >> Yakov that by default we should throw an exception and have some kind of
> >> flag (on cache or on TX?) to be able to explicitly enable this behavior.
> >>
> >
> >
> > Agree, this sounds like a good idea.
>
>

Re: IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

Savepoints marked for 2.1, exceptions for 2.0. Do you want me to make
exceptions first?

2017-03-29 11:24 GMT+03:00 Дмитрий Рябов :

> Finish savepoints or flag&exceptions for atomic operations?
> Not sure about savepoints. Exceptions - yes. https://issues.apache.
> org/jira/browse/IGNITE-2313 isn't it?
>
> 2017-03-29 2:12 GMT+03:00 Denis Magda :
>
>> If we want to make the exception based approach the default one then the
>> task has to be released in 2.0.
>>
>> Dmitriy Ryabov, do you think you can finish it (dev, review, QA) by the
>> code freeze data (April 14)?
>>
>> —
>> Denis
>>
>> > On Mar 28, 2017, at 11:57 AM, Dmitriy Setrakyan 
>> wrote:
>> >
>> > On Tue, Mar 28, 2017 at 11:54 AM, Sergi Vladykin <
>> sergi.vlady...@gmail.com>
>> > wrote:
>> >
>> >> I think updating an Atomic cache from within a transaction perfectly
>> makes
>> >> sense. For example for some kind of operations logging and so forth.
>> Still
>> >> I agree that this can be error prone and forbidden by default. I agree
>> with
>> >> Yakov that by default we should throw an exception and have some kind
>> of
>> >> flag (on cache or on TX?) to be able to explicitly enable this
>> behavior.
>> >>
>> >
>> >
>> > Agree, this sounds like a good idea.
>>
>>
>

IGNITE-4188, thoughts about realisation

[Дмитрий Рябов]

Hello, community.

try (Transaction tx = ignite.transactions().txStart()) {
c.put(1, 1);

tx.savepoint("mysavepoint");

c.put(2, 2);

tx.rollbackToSavepoint("mysavepoint");

c.put(3, 3);

tx.commit();
}


I want to make it in such way:

When user calls tx.savepoint(""), Transaction delegate call to
IgniteInternalTx, which delegate it to its IgniteTxState.

IgniteTxState implementation will contain stack of savepoints, where
Savepoint is class containing snapshot of cache entries and any important
additional information.

What do you think?

IGNITE-4188: Savepoints support inside of Ignite Transactions, thoughts about realisation

[Дмитрий Рябов]

Hello, community. Sam, Yakov, Alex G., can you look my decision?

try (Transaction tx = ignite.transactions().txStart()) {
c.put(1, 1);

tx.savepoint("mysavepoint");

c.put(2, 2);

tx.rollbackToSavepoint("mysavepoint");

c.put(3, 3);

tx.commit();
}


I want to make it in such way:

When user calls tx.savepoint(""), Transaction delegate call to
IgniteInternalTx, which delegate it to its IgniteTxState.

IgniteTxState implementation will contain stack of savepoints, where
Savepoint is class containing snapshot of cache entries and any important
additional information.

What do you think?

Re: Ignite-4795 - ready for review

[Дмитрий Рябов]

Andrey, I made fix, can you look it again?

2017-03-28 20:40 GMT+03:00 Andrey Gura :

> Dmitry, see JIRA ticket for review comments.
>
> On Mon, Mar 27, 2017 at 6:13 PM, Denis Magda  wrote:
> > Hello Dmitriy, thanks! Someone will have a look at your changes soon.
> Sorry for the delay.
> >
> > —
> > Denis
> >
> >> On Mar 27, 2017, at 3:24 AM, Дмитрий Рябов 
> wrote:
> >>
> >> Hello, can anyone review this issue?
> >>
> >> 2017-03-20 16:33 GMT+03:00 Дмитрий Рябов :
> >>
> >>> Done.
> >>>
> >>> 2017-03-20 16:30 GMT+03:00 Антон Чураев :
> >>>
> >>>> Dmitry, thank you!
> >>>>
> >>>> Could you please also change issue status to "Patch available".
> >>>>
> >>>> 2017-03-20 16:01 GMT+03:00 Дмитрий Рябов :
> >>>>
> >>>>> Hello, community. Please, review and/or suggest something about
> >>>> javadocs of
> >>>>> transactions.
> >>>>>
> >>>>> PR: https://github.com/apache/ignite/pull/1630/files
> >>>>>
> >>>>> JIRA: https://issues.apache.org/jira/browse/IGNITE-4795
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> С уважением,
> >>>> Чураев Антон
> >>>>
> >>>
> >>>
> >
>

IGNITE-2313 - ready for review

[Дмитрий Рябов]

Hello, please review.

PR: https://github.com/apache/ignite/pull/1709/files
JIRA: https://issues.apache.org/jira/browse/IGNITE-2313
CI:
http://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests_RatJavadoc&branch_IgniteTests=pull%2F1709%2Fhead&tab=buildTypeStatusDiv

Re: IGNITE-2313 - ready for review

[Дмитрий Рябов]

Em... sorry, always forgot about meaningful title(

Discussion? Yeah, I'll do it.

2017-04-05 22:15 GMT+03:00 Denis Magda :

> Dmitriy R., may I ask you to make a title and description of review
> request more meaningful? Looking at email's subject and body neither me nor
> anyone else have an idea what you ask to review. As a result most of
> community members will simply ignore your review request skipping the
> message. Don’t force people to go to JIRA ;)
>
> Igniters,
>
> This is the ticket to be reviewed: "Need to add a mode to fail atomic
> operations within a transaction”. Please review it.
>
> Dmitriy R., there was a relevant discussion around the ticket. Would you
> paste a link to it there?
>
> —
> Denis
>
>
> > On Apr 5, 2017, at 4:12 AM, Дмитрий Рябов  wrote:
> >
> > Hello, please review.
> >
> > PR: https://github.com/apache/ignite/pull/1709/files
> > JIRA: https://issues.apache.org/jira/browse/IGNITE-2313
> > CI:
> > http://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests_RatJavadoc&branch_IgniteTests=pull%2F1709%
> 2Fhead&tab=buildTypeStatusDiv
>
>

Re: IGNITE-4188, savepoints with atomic cache?

[Дмитрий Рябов]

IGNITE-2313 done, can you review it?

PR: https://github.com/apache/ignite/pull/1709/files
JIRA: https://issues.apache.org/jira/browse/IGNITE-2313
CI: http://ci.ignite.apache.org/viewType.html?buildTypeId=
IgniteTests_RatJavadoc&branch_IgniteTests=pull%2F1709%
2Fhead&tab=buildTypeStatusDiv

2017-03-29 20:58 GMT+03:00 Denis Magda :

> Sorry, I get lost in tickets.
>
> Yes, IGNITE-2313 has to be completed in 2.0 if we want to makes this
> change.
>
> —
> Denis
>
> > On Mar 29, 2017, at 2:12 AM, Дмитрий Рябов 
> wrote:
> >
> > Savepoints marked for 2.1, exceptions for 2.0. Do you want me to make
> > exceptions first?
> >
> > 2017-03-29 11:24 GMT+03:00 Дмитрий Рябов :
> >
> >> Finish savepoints or flag&exceptions for atomic operations?
> >> Not sure about savepoints. Exceptions - yes. https://issues.apache.
> >> org/jira/browse/IGNITE-2313 isn't it?
> >>
> >> 2017-03-29 2:12 GMT+03:00 Denis Magda :
> >>
> >>> If we want to make the exception based approach the default one then
> the
> >>> task has to be released in 2.0.
> >>>
> >>> Dmitriy Ryabov, do you think you can finish it (dev, review, QA) by the
> >>> code freeze data (April 14)?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On Mar 28, 2017, at 11:57 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> >>> wrote:
> >>>>
> >>>> On Tue, Mar 28, 2017 at 11:54 AM, Sergi Vladykin <
> >>> sergi.vlady...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> I think updating an Atomic cache from within a transaction perfectly
> >>> makes
> >>>>> sense. For example for some kind of operations logging and so forth.
> >>> Still
> >>>>> I agree that this can be error prone and forbidden by default. I
> agree
> >>> with
> >>>>> Yakov that by default we should throw an exception and have some kind
> >>> of
> >>>>> flag (on cache or on TX?) to be able to explicitly enable this
> >>> behavior.
> >>>>>
> >>>>
> >>>>
> >>>> Agree, this sounds like a good idea.
> >>>
> >>>
> >>
>
>

Ignite-4795 - ready for review (Inherit TransactionException and update Javadoc)

[Дмитрий Рябов]

Hello, igniters. Please, review.

PR: https://github.com/apache/ignite/pull/1630/files

JIRA: https://issues.apache.org/jira/browse/IGNITE-4795

GNITE-2313 - ready for review (fail atomic operations within a transaction)

[Дмитрий Рябов]

Hello, igniters. Can you review it? We should finish it before 2.0.

PR: https://github.com/apache/ignite/pull/1709/files
JIRA: https://issues.apache.org/jira/browse/IGNITE-2313

Re: Ignite-4795 - ready for review (Inherit TransactionException and update Javadoc)

[Дмитрий Рябов]

Andrey, actual PR is 1630. PR 1631 was created by mistake and I closed it
immediately after creating (couse ticket must have only 1 PR, isn't it?).
So ticket has only one attached link to "GitHub Pull Request #1630".

2017-04-10 14:24 GMT+03:00 Andrey Gura :

> Dmitry,
>
> this review is in progress. But I'm confused about PR number because
> in JIRA ticket we discussed PR 1631. What is actual PR number for
> latest changes?
>
> On Mon, Apr 10, 2017 at 11:38 AM, Дмитрий Рябов 
> wrote:
> > Hello, igniters. Please, review.
> >
> > PR: https://github.com/apache/ignite/pull/1630/files
> >
> > JIRA: https://issues.apache.org/jira/browse/IGNITE-4795
>

Question about cache and transaction on different nodes

[Дмитрий Рябов]

Hello, igniters. I start the node and create a transactional cache on it,
on the other node I start the transaction and "put" in previously created
cache and rollback transaction. So what should I get? Value stored before
transaction or inside rolled transaction?

public void testRollback() throws Exception {
startGrid(0);
startGrid(1);
IgniteCache cache1 = grid( 1).cache(null);
cache1.put(1, 1);
try (Transaction tx = grid(0).transactions().txStart(PESSIMISTIC,
READ_COMMITTED)) {
cache1.put(1, );
tx.rollback();
}
assertEquals((Integer) 1, cache1.get(1));
}


The question is why I got  instead of 1? If it is right behaviour - why
it nowhere explained?

Re: Question about cache and transaction on different nodes

[Дмитрий Рябов]

Why not? I do something with cache inside transaction. The only reason to
not rollback is another node?

2017-04-12 19:52 GMT+03:00 Andrey Mashenkov :

> Hi Dmitry,
>
> Looks like you start transaction on node "grid(0)", but update value on
> another node "grid(1)".
> So, technically, it is not nested transactions, right?
>
> On Wed, Apr 12, 2017 at 7:32 PM, Дмитрий Рябов 
> wrote:
>
> > Hello, igniters. I start the node and create a transactional cache on it,
> > on the other node I start the transaction and "put" in previously created
> > cache and rollback transaction. So what should I get? Value stored before
> > transaction or inside rolled transaction?
> >
> > public void testRollback() throws Exception {
> > startGrid(0);
> > startGrid(1);
> > IgniteCache cache1 = grid( 1).cache(null);
> > cache1.put(1, 1);
> > try (Transaction tx = grid(0).transactions().txStart(PESSIMISTIC,
> READ_COMMITTED)) {
> > cache1.put(1, );
> > tx.rollback();
> > }
> > assertEquals((Integer) 1, cache1.get(1));
> > }
> >
> >
> > The question is why I got  instead of 1? If it is right behaviour -
> > why it nowhere explained?
> >
> >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

When cache topology can be null?

[Дмитрий Рябов]

One case is when node is down. Is there other cases?

[Review] IGNITE-4188: Savepoints support inside of Ignite Transactions

[Дмитрий Рябов]

Hello, igniters. Can you look it? My solution is to save a snapshot of
txState when user calls tx.savepoint("id") and replace current txState with
saved state when user calls tx.rollbackToSavepoint("id").

JIRA: https://issues.apache.org/jira/browse/IGNITE-4188

PR: https://github.com/apache/ignite/pull/1815/files

Re: "Apache Ignite University”

[Дмитрий Рябов]

Yep, we need it.

2017-04-20 23:48 GMT+03:00 Prachi Garg :

> Great idea! +1
>
> -P
>
> On Thu, Apr 20, 2017 at 1:35 PM, Tom Diederich  >
> wrote:
>
> > I’d also like to propose an enhancement to the community that I believe
> > would be helpful to new and experienced Igniters alike: "Apache Ignite
> > University.”
> >
> > It would be a new section of the Ignite website that features how-to’s in
> > the form of blog posts, videos, even promoting "local" (wherever you
> might
> > be) meetups and events. The goal is to help people from the very beginner
> > to highly advanced with info and conversations.
> >
> > From "Getting started with Apache Ignite" all the way to the most
> advanced
> > topics and tasks.
> >
> > What does everyone think? Yes or no?
> >
> > -Tom
> >
> >
> >
> >
> >
> >
> >
> >
>

IGNITE-4437 Make sure data structures do not use outTx call

[Дмитрий Рябов]

Hello, igniters. Seems like someone already cut all outTx calls, because
2.0rc2 doesn't have any of them. So we can just close this ticket.

[Review] IGNITE-4648: IgniteInternalTx.prepare() does not wait for async operations to complete

[Дмитрий Рябов]

Hello, igniters, please review.

JIRA: https://issues.apache.org/jira/browse/IGNITE-4648

PR: https://github.com/apache/ignite/pull/1876/files

CI:
http://ci.ignite.apache.org/viewLog.html?buildId=590583&tab=buildResultsDiv&buildTypeId=IgniteTests_RunAllTemp

Re: waiting for partition release question

[Дмитрий Рябов]

May be let second node to finish join and enter the ring, but start
rebalance after all lock will be released.

2017-05-19 12:30 GMT+03:00 ALEKSEY KUZNETSOV :

> If we acquired a lock and a new node is joining cluster, should it wait for
> lock release?
> May be it could proceed joining ?
> The question comes from my ticket
> https://issues.apache.org/jira/browse/IGNITE-2671
>
> чт, 18 мая 2017 г. в 20:05, Alexey Goncharuk :
>
> > Hi Aleksey,
> >
> > The main purpose of this method is to wait for all ongoing updates
> > (transactional and atomic), initiated on the previous topology version,
> to
> > finish to prevent inconsistencies during rebalancing and to prevent two
> > different simultaneous owners of the same lock.
> >
> > We will be adding documentation pages on Apache Ignite wiki which will
> > explain transactions mechanics in greater detail.
> >
> > Hope this helps,
> > AG
> >
> > 2017-05-18 16:50 GMT+03:00 ALEKSEY KUZNETSOV :
> >
> > > Hi Igntrs!
> > > What is the point of waiting partition release in the end of
> > > GridDhtPartitionsExchangeFuture#init() method ?
> > > In what scenarious do we need it ?
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>

Re: [Review] IGNITE-4188: Savepoints support inside of Ignite Transactions

[Дмитрий Рябов]

Can anyone review it?

Upsource here - http://reviews.ignite.apache.org/ignite/review/IGNT-CR-186

2017-04-20 1:30 GMT+03:00 Denis Magda :

> Hi Dmitriy,
>
> Thanks a lot for the contribution. Please move the ticket to the patch
> available state.
>
> Our Tx gurus like Sam or Alex G. should be able to review after the
> community finalizes all Apache Ignite 2.0 related activity. We’re pretty
> packed with this release for now, sorry for the delay.
>
> —
> Denis
>
> > On Apr 19, 2017, at 3:23 AM, Дмитрий Рябов 
> wrote:
> >
> > Hello, igniters. Can you look it? My solution is to save a snapshot of
> > txState when user calls tx.savepoint("id") and replace current txState
> with
> > saved state when user calls tx.rollbackToSavepoint("id").
> >
> > JIRA: https://issues.apache.org/jira/browse/IGNITE-4188
> >
> > PR: https://github.com/apache/ignite/pull/1815/files
>
>

Ready for review - IGNITE-2313 (fail atomic operations within a transaction)

[Дмитрий Рябов]

Hello, please review.

PR: https://github.com/apache/ignite/pull/1709/files
JIRA: https://issues.apache.org/jira/browse/IGNITE-2313
CI:
http://ci.ignite.apache.org/viewLog.html?buildId=675135&tab=buildResultsDiv&buildTypeId=Ignite20Tests_RunAll

Re: daemon nodes

[Дмитрий Рябов]

Some messages don't go through clients. May be some daemon realization will
need them?

2017-06-28 11:22 GMT+03:00 Dmitriy Govorukhin 
:

> Dmitriy,
>
> Why daemon node may be server or client? Why it is not just daemon? I
> guess if daemon " do not store data or
> execute computations " it must be always client daemon. For me, daemon
> server, very strange.
>
> On Wed, Jun 28, 2017 at 1:25 AM, Dmitriy Setrakyan
>  wrote:
> > Daemon nodes are nodes that join the cluster, but do not store data or
> > execute computations or perform any other of the Ignite API-based
> > functionality. Visor node is a good example of a daemon node, because it
> > wants to join the cluster and receive metrics from other nodes, but does
> > not need to participate in computation or data related functionality.
> >
> > D.
> >
> > On Tue, Jun 27, 2017 at 2:49 AM, Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com> wrote:
> >
> >> Hi Igniters,
> >>
> >> Can somebody explain what does mean  daemon node? I thought that exist
> >> only client and server node, but node may can be daemon (server daemon
> >> or client daemon).  As is can see, command line visor join in topology
> >> as server daemon. Can this be an outdated functionality?
> >>
>

Re: daemon nodes

[Дмитрий Рябов]

Dmitriy Govorukhin, ServerImpl
process TcpDiscoveryJoinRequestMessage,
TcpDiscoveryStatusCheckMessage, TcpDiscoveryDiscardMessage
and TcpDiscoveryRingLatencyCheckMessage while ClientImpl
process TcpDiscoveryPingRequest. Thats the difference I see between server
and client node message workers.

2017-06-28 13:52 GMT+03:00 Dmitry Pavlov :

> Hi,
> ok, if we have 3 type of nodes: - clients - servers - daemons. Let us merge
> all types of daemons in one daemon type and remove unused code for server
> daemon. Are there any concerns about this idea?
>
> Best Regards,
> Dmitry Pavlov
>
> ср, 28 июн. 2017 г. в 12:19, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com>:
>
> > What kind of messages are you talking about? Please, provide example.
> >
> > On Wed, Jun 28, 2017 at 11:54 AM, Дмитрий Рябов 
> > wrote:
> > > Some messages don't go through clients. May be some daemon realization
> > will
> > > need them?
> > >
> > > 2017-06-28 11:22 GMT+03:00 Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com>
> > > :
> > >
> > >> Dmitriy,
> > >>
> > >> Why daemon node may be server or client? Why it is not just daemon? I
> > >> guess if daemon " do not store data or
> > >> execute computations " it must be always client daemon. For me, daemon
> > >> server, very strange.
> > >>
> > >> On Wed, Jun 28, 2017 at 1:25 AM, Dmitriy Setrakyan
> > >>  wrote:
> > >> > Daemon nodes are nodes that join the cluster, but do not store data
> or
> > >> > execute computations or perform any other of the Ignite API-based
> > >> > functionality. Visor node is a good example of a daemon node,
> because
> > it
> > >> > wants to join the cluster and receive metrics from other nodes, but
> > does
> > >> > not need to participate in computation or data related
> functionality.
> > >> >
> > >> > D.
> > >> >
> > >> > On Tue, Jun 27, 2017 at 2:49 AM, Dmitriy Govorukhin <
> > >> > dmitriy.govoruk...@gmail.com> wrote:
> > >> >
> > >> >> Hi Igniters,
> > >> >>
> > >> >> Can somebody explain what does mean  daemon node? I thought that
> > exist
> > >> >> only client and server node, but node may can be daemon (server
> > daemon
> > >> >> or client daemon).  As is can see, command line visor join in
> > topology
> > >> >> as server daemon. Can this be an outdated functionality?
> > >> >>
> > >>
> >
>

Re: [Review] IGNITE-4648: IgniteInternalTx.prepare() does not wait for async operations to complete

[Дмитрий Рябов]

I added test as Yakov said, here is CI
only test method added, without bugfix:
http://ci.ignite.apache.org/viewLog.html?buildId=694784&buildTypeId=Ignite20Tests_RunAll&tab=buildResultsDiv
and with bugfix:
http://ci.ignite.apache.org/viewLog.html?buildId=695030&buildTypeId=Ignite20Tests_RunAll&tab=buildResultsDiv

2017-05-03 13:04 GMT+03:00 Дмитрий Рябов :

> Hello, igniters, please review.
>
> JIRA: https://issues.apache.org/jira/browse/IGNITE-4648
>
> PR: https://github.com/apache/ignite/pull/1876/files
>
> CI: http://ci.ignite.apache.org/viewLog.html?buildId=
> 590583&tab=buildResultsDiv&buildTypeId=IgniteTests_RunAllTemp
>

Prepare works as it should work in JTA

[Дмитрий Рябов]

Hi, guys. I've done IGNITE-4648, the idea was to improve prepare
implementations, because async calls could hangs or fail if transaction
state changes were not finished [1].

I modified internal transaction's prepare implementation, so now it wait
for unfinished futures in transaction and added test to check improvement.
Also Pavel Tupitsyn updated .NET tests to check this fix.

Nothing changed for Ignite users, because public API wasn't changed.

Completing this task decreased chances of unexpected behavior in async
JTA-transaction.

But we still need more test coverage, anyone want to contribute some tests
[2]?

[1] https://issues.apache.org/jira/browse/IGNITE-4648
[2] https://issues.apache.org/jira/browse/IGNITE-5748

Re: Set default cache synchronization mode to FULL_SYNC

[Дмитрий Рябов]

+1 to change PRIMARY_SYNC to FULL_SYNC.

I think it is not reasonable to set readFromBackup=true by default,
especially for replicated caches, but FULL_SYNCE will keep cache in
consistent state.

2017-08-04 13:23 GMT+03:00 Anton Vinogradov :

> +1 to change PRIMARY_SYNC to FULL_SYNC and keep readFromBackup=true
>
> Dmitriy,
> Why should we wait for 3.0?
> This change looks safe for me.
>
> On Wed, Aug 2, 2017 at 9:51 PM, Dmitriy Setrakyan 
> wrote:
>
> > We have to wait with any default changes to 3.0, unfortunately.
> >
> > On Wed, Aug 2, 2017 at 8:30 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Not sure about readFromBackup, but changing PRIMARY_SYNC to FULL_SYNC
> > looks
> > > safe to me. Any other thoughts?
> > >
> > > ср, 2 авг. 2017 г. в 21:10, Denis Magda :
> > >
> > > > +1 for both suggestions but I’m not sure we can do the change till
> 3.0.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Aug 2, 2017, at 1:27 AM, Vladimir Ozerov 
> > > > wrote:
> > > > >
> > > > > +1 for readFromBackup=false as well :-) Another example of default
> > > value
> > > > > with subtle effects.
> > > > >
> > > > > On Wed, Aug 2, 2017 at 11:11 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com> wrote:
> > > > >
> > > > >> Vladimir,
> > > > >>
> > > > >> Personally, I agree that we should put correctness over
> performance,
> > > > >> however (1) is not a correct statement for TRANSACTIONAL caches. A
> > > > >> transactional client always validates the result of an operation
> and
> > > > throw
> > > > >> a correct exception if operation failed. (1) is true for ATOMIC
> > > caches,
> > > > >> though.
> > > > >>
> > > > >> A user can get in trouble in this default for both TX and ATOMIC
> > > caches
> > > > if
> > > > >> a put is performed from a backup node and readFromBackup is set to
> > > > false.
> > > > >> In this case, the simple read-after-write scenario may fail. I
> would
> > > > rather
> > > > >> set readFromBackup to false by default, however, this fixes
> neither
> > > the
> > > > SQL
> > > > >> nor ATOMIC caches issues.
> > > > >>
> > > > >> +1 for the change, and extend the warning for partitioned caches
> > with
> > > > >> readFromBackup=true and PRIMARY_SYNC.
> > > > >>
> > > > >>
> > > > >>
> > > > >> 2017-08-02 10:58 GMT+03:00 Vladimir Ozerov  >:
> > > > >>
> > > > >>> Igniters,
> > > > >>>
> > > > >>> I want to re-iterate idea of changing default synchronization
> mode
> > > from
> > > > >>> PRIMARY_SYNC to FULL_SYNC.
> > > > >>>
> > > > >>> Motivation:
> > > > >>> 1) If user set [cacheMode=PARTITIONED, backups=1] he still could
> > > loose
> > > > >> data
> > > > >>> silently. Because primary node could report success to the client
> > and
> > > > >> then
> > > > >>> crash before data is propagated to backups.
> > > > >>> 2) If user set [cacheMode=REPLICATED] and use SQL, he will might
> > get
> > > > >>> invalid results if cache is being updated concurrently - well
> known
> > > > >> issue.
> > > > >>>
> > > > >>> The only advantage of PRIMARY_SYNC is slightly better
> performance,
> > > but
> > > > we
> > > > >>> should prefer correctness over performance.
> > > > >>>
> > > > >>> Proposed changes:
> > > > >>> 1) Make FULL_SYNC default;
> > > > >>> 2) Print a warning about possibly incorrect SQL results if
> > REPLICATED
> > > > >> cache
> > > > >>> is started in PRIMARY_SYNC mode.
> > > > >>>
> > > > >>> Thoughts?
> > > > >>>
> > > > >>> Vladimir.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Set default cache synchronization mode to FULL_SYNC

[Дмитрий Рябов]

I mean "set readFromBackup = false" (copy-paste was bad idea).

2017-08-04 14:21 GMT+03:00 Дмитрий Рябов :

> +1 to change PRIMARY_SYNC to FULL_SYNC.
>
> I think it is not reasonable to set readFromBackup=true by default,
> especially for replicated caches, but FULL_SYNCE will keep cache in
> consistent state.
>
> 2017-08-04 13:23 GMT+03:00 Anton Vinogradov :
>
>> +1 to change PRIMARY_SYNC to FULL_SYNC and keep readFromBackup=true
>>
>> Dmitriy,
>> Why should we wait for 3.0?
>> This change looks safe for me.
>>
>> On Wed, Aug 2, 2017 at 9:51 PM, Dmitriy Setrakyan 
>> wrote:
>>
>> > We have to wait with any default changes to 3.0, unfortunately.
>> >
>> > On Wed, Aug 2, 2017 at 8:30 PM, Vladimir Ozerov 
>> > wrote:
>> >
>> > > Not sure about readFromBackup, but changing PRIMARY_SYNC to FULL_SYNC
>> > looks
>> > > safe to me. Any other thoughts?
>> > >
>> > > ср, 2 авг. 2017 г. в 21:10, Denis Magda :
>> > >
>> > > > +1 for both suggestions but I’m not sure we can do the change till
>> 3.0.
>> > > >
>> > > > —
>> > > > Denis
>> > > >
>> > > > > On Aug 2, 2017, at 1:27 AM, Vladimir Ozerov > >
>> > > > wrote:
>> > > > >
>> > > > > +1 for readFromBackup=false as well :-) Another example of default
>> > > value
>> > > > > with subtle effects.
>> > > > >
>> > > > > On Wed, Aug 2, 2017 at 11:11 AM, Alexey Goncharuk <
>> > > > > alexey.goncha...@gmail.com> wrote:
>> > > > >
>> > > > >> Vladimir,
>> > > > >>
>> > > > >> Personally, I agree that we should put correctness over
>> performance,
>> > > > >> however (1) is not a correct statement for TRANSACTIONAL caches.
>> A
>> > > > >> transactional client always validates the result of an operation
>> and
>> > > > throw
>> > > > >> a correct exception if operation failed. (1) is true for ATOMIC
>> > > caches,
>> > > > >> though.
>> > > > >>
>> > > > >> A user can get in trouble in this default for both TX and ATOMIC
>> > > caches
>> > > > if
>> > > > >> a put is performed from a backup node and readFromBackup is set
>> to
>> > > > false.
>> > > > >> In this case, the simple read-after-write scenario may fail. I
>> would
>> > > > rather
>> > > > >> set readFromBackup to false by default, however, this fixes
>> neither
>> > > the
>> > > > SQL
>> > > > >> nor ATOMIC caches issues.
>> > > > >>
>> > > > >> +1 for the change, and extend the warning for partitioned caches
>> > with
>> > > > >> readFromBackup=true and PRIMARY_SYNC.
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> 2017-08-02 10:58 GMT+03:00 Vladimir Ozerov > >:
>> > > > >>
>> > > > >>> Igniters,
>> > > > >>>
>> > > > >>> I want to re-iterate idea of changing default synchronization
>> mode
>> > > from
>> > > > >>> PRIMARY_SYNC to FULL_SYNC.
>> > > > >>>
>> > > > >>> Motivation:
>> > > > >>> 1) If user set [cacheMode=PARTITIONED, backups=1] he still could
>> > > loose
>> > > > >> data
>> > > > >>> silently. Because primary node could report success to the
>> client
>> > and
>> > > > >> then
>> > > > >>> crash before data is propagated to backups.
>> > > > >>> 2) If user set [cacheMode=REPLICATED] and use SQL, he will might
>> > get
>> > > > >>> invalid results if cache is being updated concurrently - well
>> known
>> > > > >> issue.
>> > > > >>>
>> > > > >>> The only advantage of PRIMARY_SYNC is slightly better
>> performance,
>> > > but
>> > > > we
>> > > > >>> should prefer correctness over performance.
>> > > > >>>
>> > > > >>> Proposed changes:
>> > > > >>> 1) Make FULL_SYNC default;
>> > > > >>> 2) Print a warning about possibly incorrect SQL results if
>> > REPLICATED
>> > > > >> cache
>> > > > >>> is started in PRIMARY_SYNC mode.
>> > > > >>>
>> > > > >>> Thoughts?
>> > > > >>>
>> > > > >>> Vladimir.
>> > > > >>>
>> > > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Дмитрий Рябов]

; > > No doubts,
> > > > > > > it would be a valuable addition to Ignite.
> > > > > > >
> > > > > > > I would encourage you creating an IEP on the wiki and break the
> > > work into
> > > > > > > pieces discussing specific part with the community.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 1, 2018 at 9:29 PM, Nikolay Izhikov <
> > > nizhi...@apache.org> wrote:
> > > > > > >
> > > > > > > > Hello, Dmitriy.
> > > > > > > >
> > > > > > > > Thank you for feedback!
> > > > > > > >
> > > > > > > > > Will it be supported?
> > > > > > > >
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > TDE shouldn't broke any of existing Ignite features.
> > > > > > > > It adds some encrypt/decrypt level when we writing and
> reading
> > > pages
> > > > > > > > in/from PDS.
> > > > > > > >
> > > > > > > > В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет:
> > > > > > > > > I have looked at the design, but could not find anything
> > about
> > > running
> > > > > > > >
> > > > > > > > SQL
> > > > > > > > > queries against the encrypted data. Will it be supported?
> > > > > > > > >
> > > > > > > > > D.
> > > > > > > > >
> > > > > > > > > On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov <
> > > nizhi...@apache.org>
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hell, Dima!
> > > > > > > > > >
> > > > > > > > > > Thank you for document!
> > > > > > > > > >
> > > > > > > > > > I'm ready to implement this feature with you.
> > > > > > > > > >
> > > > > > > > > > Igniters, please, share you thoughts about proposed
> design
> > > > > > > > > >
> > > > > > > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > > > > > > > > >
> > > > > > > > > > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > > > > > > > > > > Hello, Igniters!
> > > > > > > > > > >
> > > > > > > > > > > I investigated the issue and wrote some details in a
> > draft
> > > document
> > > > > > > > > > > [1]. I think we should made IEP for TDE because it is a
> > > big change
> > > > > > > >
> > > > > > > > and
> > > > > > > > > > > should be described in a single place, but not in a
> > message
> > > > > > > > > > > conversation.
> > > > > > > > > > > Please, look it and write your thoughts. What is not
> > > understandable,
> > > > > > > > > > > what should be detailed or described?
> > > > > > > > > > >
> > > > > > > > > > > > Where are we going to store keys (MEK) physically?
> > Would
> > > it be
> > > > > > > >
> > > > > > > > PKCS#11
> > > > > > > > > > > > storage? Where we will store passwords to unlock
> > storage
> > > or it
> > > > > > > >
> > > > > > > > will be
> > > > > > > > > > > > responibilty of user?
> > > > > > > > > > >
> > > > > > > > > > > I think we should provide interface for MEK storage to
> > let
> > > users use
> > > > > > > > > > > storages they want. I suppose at the first step we
> should
> > > provide
> > > > > > > >
> > > > > > > > very
> > > > > > > > > > > simple implementation, which will store MEK on every
> node
> > > and MEK
> > > > > > > >
> > > > > > > > will
> > > > > > > > > > > be extracted by administrator during cluster activation
> > > process. Once
> > > > > > > > > > > MEK is extracted from key store, we decrypt CEKs and
> > > destroy open
> > > > > > > >
> > > > > > > > MEK,
> > > > > > > > > > > leaving open only cache keys.
> > > > > > > > > > >
> > > > > > > > > > > I think external storage is user's worry and we
> shouldn't
> > > give users
> > > > > > > > > > > built-in external storage like Oracle Wallet or
> Microsoft
> > > Azure Key
> > > > > > > > > > > Vault because it will increase Ignite's complexity too
> > > much.
> > > > > > > > > > >
> > > > > > > > > > > And yes, we should to comply with the standards like
> > > PKCS#11.
> > > > > > > > > > >
> > > > > > > > > > > > One more thing is how "node gets MEK from
> coordinator",
> > > if we send
> > > > > > > > > > > > cleartext MEK, such security becomes useless also.
> > > > > > > > > > >
> > > > > > > > > > > Yeah, that's why we should use secured connection. As I
> > > know, we have
> > > > > > > > > > > SSL implementation over JDK implementation, am I right?
> > > But we must
> > > > > > > > > > > ensure to use latest SSL/TLS version.
> > > > > > > > > > >
> > > > > > > > > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > >
> >
>

Re: IGNITE-6879

[Дмитрий Рябов]

Dmitry, Alexey, code and tests looks good for me.

JIRA: https://issues.apache.org/jira/browse/IGNITE-6879
PR: https://github.com/apache/ignite/pull/3704
Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-541
TeamCity:
https://ci.ignite.apache.org/viewLog.html?buildId=1170373&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_RunAll

2018-03-22 21:48 GMT+03:00 Роман Меерсон :

> Hi all!
>
> Dmitriy thank you for review.
>
> I`ve just fixed all your comments.
>
> чт, 22 мар. 2018 г. в 20:36, Dmitry Pavlov :
>
>> HI Dmitriy, thank you!
>>
>> Roman, could you please address Dmitriy's comments?
>>
>> чт, 22 мар. 2018 г. в 19:18, Дмитрий Рябов :
>>
>>> Hi Dmitriy,
>>>
>>> I took a look for PR, it needs codestyle fixes.
>>>
>>> 2018-03-19 14:22 GMT+03:00 Dmitry Pavlov :
>>>
>>>> Hi Alexey,
>>>>
>>>> Did you find the patch is looking good and is ready to be merged?
>>>>
>>>> Sincerely,
>>>> Dmitriy Pavlov
>>>>
>>>> чт, 15 мар. 2018 г. в 11:19, Alexey Kukushkin <
>>>> kukushkinale...@gmail.com>:
>>>>
>>>> > Just found the fix is ready - I will review it today or tomorrow.
>>>> >
>>>>
>>>
>>>

Re: IGNITE-6879

[Дмитрий Рябов]

Hi!
No, it wasn't a joke)

2018-04-04 11:22 GMT+03:00 Роман Меерсон :

> Hi all!
> I suppose Dmity`s message wasn`t april 1st joke)
> So what about my PR? Would it be merged?
>
> вс, 1 апр. 2018 г. в 12:40, Дмитрий Рябов :
>
>> Dmitry, Alexey, code and tests looks good for me.
>>
>> JIRA: https://issues.apache.org/jira/browse/IGNITE-6879
>> PR: https://github.com/apache/ignite/pull/3704
>> Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-541
>> TeamCity: https://ci.ignite.apache.org/viewLog.html?buildId=1170373&tab=
>> buildResultsDiv&buildTypeId=IgniteTests24Java8_RunAll
>>
>> 2018-03-22 21:48 GMT+03:00 Роман Меерсон :
>>
>>> Hi all!
>>>
>>> Dmitriy thank you for review.
>>>
>>> I`ve just fixed all your comments.
>>>
>>> чт, 22 мар. 2018 г. в 20:36, Dmitry Pavlov :
>>>
>>>> HI Dmitriy, thank you!
>>>>
>>>> Roman, could you please address Dmitriy's comments?
>>>>
>>>> чт, 22 мар. 2018 г. в 19:18, Дмитрий Рябов :
>>>>
>>>>> Hi Dmitriy,
>>>>>
>>>>> I took a look for PR, it needs codestyle fixes.
>>>>>
>>>>> 2018-03-19 14:22 GMT+03:00 Dmitry Pavlov :
>>>>>
>>>>>> Hi Alexey,
>>>>>>
>>>>>> Did you find the patch is looking good and is ready to be merged?
>>>>>>
>>>>>> Sincerely,
>>>>>> Dmitriy Pavlov
>>>>>>
>>>>>> чт, 15 мар. 2018 г. в 11:19, Alexey Kukushkin <
>>>>>> kukushkinale...@gmail.com>:
>>>>>>
>>>>>> > Just found the fix is ready - I will review it today or tomorrow.
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>

Re: IGNITE-6879

[Дмитрий Рябов]

I agree that increasing complexity isn't good idea. Roman, can you document
the migration guide?

2018-04-04 13:41 GMT+03:00 Alexey Kukushkin :

> Roman, Dmitry,
>
> I also reviewed the fix and the code looks OK to me. But the fix has
> significant implication - Ignite no longer can be used with spring-data 1.0
> due to no backward compatibility between spring 2.0 and 1.0 APIs. With this
> approach we must remember to add corresponding spring-data migration
> instructions to the future ignite 2.5 migration guide.
>
> We could keep spring 1 support and backward compatibility by creating a new
> module "ignite-spring-2-data" and keeping existing ignite-spring-data as
> is. I do not like this option since to me increased complexity and
> maintainability costs overweight the benefits of protecting
> "Ignite-spring-1" users.
>
> I suggest you find a committer (see this list
> ), communicate
> the implication I mentioned above and say that two people already approved
> the code providing we are OK with the chosen approach.
>

Re: Refactoring of IgniteKernal ack methods

[Дмитрий Рябов]

IgniteKernal and some other classes have more places to refactor. At least
for better readable form or to remove inner class [1]. May be create some
tickets for refactoring such complex places? I mean to create main ticket
like [2] or [3], and when someone see something complex *and* the way to
refactor - he can create subtask/PR for refactoring. Thoughts?

[1] - http://apache-ignite-developers.2346864.n4.nabble.com/Mini
mize-amount-of-inner-classes-predicates-tuples-etc-td17689.html
[2] - https://issues.apache.org/jira/browse/IGNITE-4575
[3] - https://issues.apache.org/jira/browse/IGNITE-5156


2017-10-11 18:37 GMT+03:00 Dmitriy Setrakyan :

> On Wed, Oct 11, 2017 at 7:44 AM, Иван Федотов  wrote:
>
> > Hello, Igniters!
> >
> > I found, that in several places of IgniteKernal class code blocks are
> huge
> > and hard to understand and in other places methods have the same context
> > and could be placed in their own class. For example methods:
> > “ackAsciiLogo”, “ackConfigUrl”, “ackDaemon”, “ackOsInfo”,
> > “ackLanguageRuntime”, “ackRemoteManagement”, “ackVmArguments”,
> > “ackClassPaths”, “ackSystemProperties”, “ackEnviromentVariables”,
> > “ackMemoryConfiguration”, “ackCacheConfiguration”, “ackP2PConfiguration”,
> > “ackRebalanceConfiguration”, which are called in 800-813 lines and
> together
> > contain over than 250 lines, can be placed in separate class
> > “AckVariousInformation”.
> >
> > Do you think that it will be good to create task on such refactoring?
> >
>
> I think this is a matter of taste. One could argue that the code is more
> readable now because these methods belong to IgniteKernal and are not
> called from any other place. I would generally such changes.
>

IGNITE-6802 is ready for review (NullPointerException when WAL store and WAL archive paths are same )

[Дмитрий Рябов]

Hello, Igniters! Please review.

JIRA: https://issues.apache.org/jira/browse/IGNITE-6802
PR: https://github.com/apache/ignite/pull/3021/files
Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-396
CI tests: 
https://ci.ignite.apache.org/viewLog.html?buildId=945125&tab=buildResultsDiv&buildTypeId=Ignite20Tests_RunAll

Re: IGNITE-4447 ready for review

[Дмитрий Рябов]

Hello.

Semyon, test passed.

PR (squashed and merged with master): https://github.com/apache/ignite/pull/3204
CI tests: 
https://ci.ignite.apache.org/viewLog.html?buildId=993439&tab=buildResultsDiv&buildTypeId=Ignite20Tests_IgniteCache5

2017-05-19 4:46 GMT+03:00 Вадим Опольский :
> Hi guys!
>
> Semyon, test suite Ignite Cache 5 with Cache150ClientsTest failed. Run it
> twice. But local it succesfully executed. What do you think about it ?
>
> http://ci.ignite.apache.org/viewLog.html?buildId=616787&tab=buildResultsDiv&buildTypeId=IgniteTests_IgniteCache5
>
> Vadim
>
> 2017-05-17 11:34 GMT+03:00 Вадим Опольский :
>
>> Hi Semyon!
>>
>> I had problem with starting TC test yesterday. Just have queued it -
>> http://ci.ignite.apache.org/project.html?projectId=IgniteTests&branch_
>> IgniteTests=pull%2F1952%2Fmerge
>>
>> Vadim
>>
>> 2017-05-16 14:10 GMT+03:00 Semyon Boikov :
>>
>>> Hi Vadim,
>>>
>>> Did you check test pass on TeamCity in this suite, could you please
>>> provide
>>> link for related TeamCity run?
>>>
>>> Thanks
>>>
>>> On Tue, May 16, 2017 at 1:35 PM, Вадим Опольский 
>>> wrote:
>>>
>>> > Hi guys!
>>> >
>>> > IgniteCache150ClientsTest.java was removed from
>>> > IgniteClientTestSuite.java to IgniteCacheTestSuite.java. Single test
>>> suite
>>> > was deleted.
>>> >
>>> > Review please this fix - https://github.com/apache/ignite/pull/1952
>>> >
>>> > Vadim Opolskii
>>> >
>>> >
>>> >
>>> > 2017-05-15 11:38 GMT+03:00 Вадим Опольский :
>>> >
>>> > > Hi guys!
>>> > >
>>> > > Vladimir, is issue https://issues.apache.org/jira/browse/IGNITE-4447
>>> > > actual now?
>>> > >
>>> > > Which tests do u mean? Detail please.
>>> > >
>>> > > Vadim
>>> > >
>>> >
>>>
>>
>>

Re: Switching Java language level to 8

[Дмитрий Рябов]

+1
Agree, I think changes will be less difficult if we go through 7/8 ->
8 -> 8/9 than 7/8 -> 7/8/9 ( -> possible 7/8/9/10 huh?) -> 8/9.

2017-12-14 14:18 GMT+03:00 Anton Vinogradov :
> +1
> Can we switch to Java8 right now and then support Java9?
> That's the easiest way I think.
>
> On Thu, Dec 14, 2017 at 1:50 PM, Andrey Kuznetsov  wrote:
>
>> Hi Igniters,
>>
>> As far as I know, we planned to get the project compilable/runnable on Java
>> 9 and then switch to Java 8 language features. But some tickets like [1]
>> require Java 8 to be fixed. It seems we are to switch source level of the
>> project to 1.8 first.
>>
>> Thoughts?
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-6734
>>
>> --
>> Best regards,
>>   Andrey Kuznetsov.
>>

IGNITE-5136 is ready for review (GridLogThrottle memory leak)

[Дмитрий Рябов]

Hello, igniters!
I've done with this ticket, tests are ok. Can you review?

JIRA: https://issues.apache.org/jira/browse/IGNITE-5136
PR: https://github.com/apache/ignite/pull/3236
Upsource (old changes):
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-344
Upsource (new changes):
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-436
CI tests: 
https://ci.ignite.apache.org/viewLog.html?buildId=1004471&tab=buildResultsDiv&buildTypeId=Ignite20Tests_RunAll

IGNITE-6826 is ready for review (Change default DiscoverySpi ipFinder type for examples)

[Дмитрий Рябов]

Hello Igniters!
I’ve changed default ipFinder in examples from multicast to static, so
examples will run faster.
Please, review.

JIRA: https://issues.apache.org/jira/browse/IGNITE-6826
GitHub: https://github.com/apache/ignite/pull/3362
Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-454

Re: [Review] IGNITE-4188: Savepoints support inside of Ignite Transactions

[Дмитрий Рябов]

Hello! Semen, can you continue to review this ticket [1]?

About my realisation:

I added `savepoint()`, `rollbackToSavepoint()` and
'releaseSavepoint()' methods to `Transaction` interface. Savepoints
are stored in `txState` of transaction on the initiator node and
contain every change made since last created savepoint or tx started
(for the first savepoint).
When user calls `savepoint()` during a transaction - its `txState`
will create savepoint with current state.
When user calls `rollbackToSavepoint()` - `txState` will replace
current state to snapshot from chosen savepoint. For pessimistic
transactions, we wait here for request/response to unlock all rolled
back keys.
When user calls 'releaseSavepoint()' - `txState` will remove snapshot
for this and consequent savepoints.

Savepoints are good if you have multiple operations inside one long
transaction and they can fail, but you still need to commit nonfailed
operations. Savepoints allow you to rollback a piece of the
transaction if you want to remain some changes. For example, we have 3
caches: "accounts" with some data, "actions" cache with data about
actions for specific accounts and "bans" with data about banned
accounts. We need to update "accounts" and "actions" caches in the
same pessimistic transaction and this update shouldn't be seen until
done. We update in next sequence: we update account first and then its
actions. After updating actions we have check that this account is
still available and not banned in the third cache (it may be banned
during our transaction). And when we see that account isn't available
– we rollback to savepoint before this account and mark it as banned
in the "account" cache instead of updating its actions.

[1] JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-4188
GitHub PR: https://github.com/apache/ignite/pull/1815
Upsource PR: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-186

2017-05-24 11:08 GMT+03:00 Дмитрий Рябов :
> Can anyone review it?
>
> Upsource here - http://reviews.ignite.apache.org/ignite/review/IGNT-CR-186
>
> 2017-04-20 1:30 GMT+03:00 Denis Magda :
>>
>> Hi Dmitriy,
>>
>> Thanks a lot for the contribution. Please move the ticket to the patch
>> available state.
>>
>> Our Tx gurus like Sam or Alex G. should be able to review after the
>> community finalizes all Apache Ignite 2.0 related activity. We’re pretty
>> packed with this release for now, sorry for the delay.
>>
>> —
>> Denis
>>
>> > On Apr 19, 2017, at 3:23 AM, Дмитрий Рябов 
>> > wrote:
>> >
>> > Hello, igniters. Can you look it? My solution is to save a snapshot of
>> > txState when user calls tx.savepoint("id") and replace current txState
>> > with
>> > saved state when user calls tx.rollbackToSavepoint("id").
>> >
>> > JIRA: https://issues.apache.org/jira/browse/IGNITE-4188
>> >
>> > PR: https://github.com/apache/ignite/pull/1815/files
>>
>

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Дмитрий Рябов]

Hi, Igniters! I think it would be nice to implement encryption in
Ignite. Even SQLite and H2 have encryption so why Ignite don't have
it?

I'd like to propose a design for discussion.

Configurations:
IgniteConfiguration:
- KeyStore tdeKeyStore - contain encryption keys.
- Encryptor encryptor - interface for encryption algorythm with
methods `encrypt()`, `decrypt()`.
CacheConfiguration:
- TransparentDataEncryption tde - configures where or how we will
encrypt. We can encrypt pages (PDS or PDS&offheap) or cached objects
(PDS, PDS&offheap and even heap if we encrypt just after key
validation and security check). And, of course, TDE can be turned off.

Encryption keys:
Every node has keystore with Cache Encryption Keys (CEKs), one key for
one cache, they are stored in encrypted form.
CEKs are encrypted by Master Encryption Key (MEK), which stored on the
coordinator or somewhere out of cluster (this way is safer). Also, we
need backups for MEK on other nodes or somewhere out of the cluster.

Encryption:
To make any operation with encrypting cache node gets MEK from
coordinator (or somewhere out of the cluster). Then it decrypts needed
CEK. With decrypted CEK we encrypt/decrypt data. When transaction
finished we must destroy received MEK.

TransparentDataEncryption enum:
NONE - no encryption.
ALL_OBJECTS - data will be encrypted on tx initiator just after key
validation and security check, so most of time it would be just a
byte[] object. It is good, but on the other hand we have additional
encryption for any node with listeners for data changing.
PDS_ONLY_OBJECTS - data will be encrypted on primary and backups when
storing to PDS.
OFFHEAP_PDS_OBJECTS - data will be encrypted when storing to offheap
storage and PDS.
PDS_ONLY_PAGES - data pages will be encrypted on primary and backups
when storing to PDS.
OFFHEAP_PDS_PAGES - data pages will be encrypted when storing to
offheap storage and PDS.

But I'm not sure about objects/pages. Do we really need both of them?


2017-06-26 14:39 GMT+03:00 Sergi Vladykin :
> No, we don't have plans for it.
>
> Sergi
>
> 2017-06-26 14:20 GMT+03:00 Vyacheslav Daradur :
>
>> Sergi, thanks for the answer.
>>
>> >> see TDE is just an option for PCI DSS compliancy but not a requirement.
>> Requirement: "Protect stored cardholder data"
>> Encryption is required.
>> TDE - is one of ways to implement it at the database level.
>>
>> Sure, an implementation at the application level solve it.
>>
>> I meant another.
>> I thought maybe this feature is in the Ignite roadmap?
>>
>>
>> 2017-06-26 13:53 GMT+03:00 Sergi Vladykin :
>>
>> > I think no one is interested in this stuff right now.
>> >
>> > Also as far as I see TDE is just an option for PCI DSS compliancy but
>> not a
>> > requirement.
>> >
>> > Your system should pass PCI DSS if you will do the required encryption at
>> > the application level and will properly manage encryption keys.
>> >
>> > Sergi
>> >
>> > 2017-06-26 11:40 GMT+03:00 Vyacheslav Daradur :
>> >
>> > > Guys, any thoughts?
>> > >
>> > > 2017-06-20 11:02 GMT+03:00 Vyacheslav Daradur :
>> > >
>> > > > Hi Igniters.
>> > > >
>> > > > I have some user cases where I need fast storage with TDE support.
>> > > > It is requered for PCI DSS certification.
>> > > >
>> > > > As far as I know AI doesn't support it.
>> > > >
>> > > > I looked at other storages.
>> > > > Many storages support it or are engaged in development this feature.
>> > > >
>> > > > Cassandra community are working on TDE support.[1]
>> > > >
>> > > > Oracle support it.[2] Moreover it supports indexing and querying on
>> > > > encrypted data.
>> > > >
>> > > > I think it will be very usefull to support TDE by AI.
>> > > >
>> > > > What do you think? Maybe development is already under way?
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/CASSANDRA-9945
>> > > > [2] https://docs.oracle.com/cd/B19306_01/network.102/b14268/
>> > > > asotrans.htm#ASOAG600
>> > > >
>> > > > --
>> > > > Best Regards, Vyacheslav D.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best Regards, Vyacheslav D.
>> > >
>> >
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>

Re: Abandoned Patch Available JIRA tickets

[Дмитрий Рябов]

Dmitry, there are 103 tickets for Cassandra [1] and 120 for Zookeeper [2].

[1] 
https://issues.apache.org/jira/browse/CASSANDRA-14132?jql=project%3Dcassandra%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d
[2] 
https://issues.apache.org/jira/issues/?jql=project%3Dzookeeper%20AND%20status%3D"Patch%20Available"%20AND%20updated%20<%20-30d

2018-02-05 15:08 GMT+03:00 Pavel Tupitsyn :
> Fixed links:
>
> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d
>
> https://issues.apache.org/jira/issues/?jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>
> On Mon, Feb 5, 2018 at 3:05 PM, Dmitry Pavlov  wrote:
>
>> Hi Denis,
>>
>> There is an interesting aspect of this issue for me.
>>
>> How close is Ignite community in this metric (not reviewed/not merged
>> tickets) to other Apache communities? For example, to Cassandra.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>>
>> пн, 5 февр. 2018 г. в 14:51, Pavel Tupitsyn :
>>
>> > Bumping the thread.
>> >
>> > Denis Magda, as a PMC chair, can you please look into this?
>> >
>> > On Tue, Jan 30, 2018 at 5:26 PM, Pavel Tupitsyn 
>> > wrote:
>> >
>> > > Igniters,
>> > >
>> > > We have *almost 60* of PATCH AVAILABLE tickets that are inactive for
>> more
>> > > than a month:
>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>> > > 20updated%20%3C%20-30d
>> > >
>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d)
>> > >
>> > >
>> > > Some of these tickets ask specific people for review without any
>> answer.
>> > > You can check if you have any pending reviews with this link:
>> > > https://issues.apache.org/jira/issues/?filter=12333285&;
>> > > jql=project%3Dignite%20AND%20status%3D%22Patch%20Available%22%20AND%
>> > > 20updated%20%3C%20-30d%20AND%20comment%20%20~%20currentUser()
>> > >
>> > > (JQL: project=ignite AND status="Patch Available" AND updated < -30d
>> AND
>> > > comment  ~ currentUser())
>> > >
>> > >
>> > > I encourage everyone to walk through these tickets and see if you can
>> > help
>> > > with reviewing and merging some of these contributions.
>> > >
>> > > Thanks,
>> > > Pavel
>> > >
>> >
>>

Re: Author in header of class

[Дмитрий Рябов]

For the first profit you should see maintainer table [1].

[1]
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers

2018-02-14 11:42 GMT+03:00 Vyacheslav Daradur :

> Hi Anton,
>
> Apache Ignite is licensed under the Apache License Version 2.0, which
> does not allow to use author tag in java code [1].
>
> [1] http://directory.apache.org/fortress/coding-standards.
> html#classinterface-headers
>
> On Wed, Feb 14, 2018 at 11:23 AM, Антон Калашников 
> wrote:
> > Hello Ignite Community!
> >
> > My name is Anton. I joined to community some time ago and I want to
> contribute to Apache Ignite.
> >
> > I would be want to make my first proposal. I noticed that Ignite don't
> have author description in header of classes unlike many other apache
> projects.
> > I propose to use javadoc tag @author in header of class when you created
> it  and also add extra tag @author when you do many changes in this class.
> >
> > Profits of this aproach, in my opinion:
> > 1) You always know who has knowledge of this class unlike git annotate
> which are sensetive in refactoring, moving, etc.
> > 2) It will highed responsibilty for code quality because nobody want to
> sign under bad code)
> >
> > Example:
> > /**
> >  * @author Anton Kalashnikov
> >  * @author Other Author
> >  */
> >
> > --
> > Best Regards,
> > Anton Kalashnikov
> >
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Дмитрий Рябов]

Hello, Igniters!

I investigated the issue and wrote some details in a draft document
[1]. I think we should made IEP for TDE because it is a big change and
should be described in a single place, but not in a message
conversation.
Please, look it and write your thoughts. What is not understandable,
what should be detailed or described?

> Where are we going to store keys (MEK) physically? Would it be PKCS#11
> storage? Where we will store passwords to unlock storage or it will be
> responibilty of user?

I think we should provide interface for MEK storage to let users use
storages they want. I suppose at the first step we should provide very
simple implementation, which will store MEK on every node and MEK will
be extracted by administrator during cluster activation process. Once
MEK is extracted from key store, we decrypt CEKs and destroy open MEK,
leaving open only cache keys.

I think external storage is user's worry and we shouldn't give users
built-in external storage like Oracle Wallet or Microsoft Azure Key
Vault because it will increase Ignite's complexity too much.

And yes, we should to comply with the standards like PKCS#11.

> One more thing is how "node gets MEK from coordinator", if we send
> cleartext MEK, such security becomes useless also.

Yeah, that's why we should use secured connection. As I know, we have
SSL implementation over JDK implementation, am I right? But we must
ensure to use latest SSL/TLS version.

[1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf

Re: Ignite contibutors page

[Дмитрий Рябов]

Hi Igniters!

I'm not mentioned too. Dmitriy Ryabov.

2018-03-05 23:39 GMT+03:00 Denis Magda :

> Hi Dmitriy,
>
> Thanks for kicking off the discussion. I'll wait for a couple of days for
> other response and update the page in bulk.
>
> --
> Denis
>
> On Mon, Mar 5, 2018 at 12:32 PM, Dmitry Pavlov 
> wrote:
>
> > Hi Igniters,
> >
> > I noticed that I'm not mentioned on the community contributors page:
> > https://ignite.apache.org/community/resources.html
> >
> > Perhaps there is someone else, who is not yet added. Or information is
> not
> > up-to-date.
> >
> > Please check the data for yourself and respond if you need to be added /
> or
> > information needs to be updated.
> >
> > Sincerely,
> > Dmitry Pavlov
> >
>

Re: IGNITE-6879

[Дмитрий Рябов]

Hi Dmitriy,

I took a look for PR, it needs codestyle fixes.

2018-03-19 14:22 GMT+03:00 Dmitry Pavlov :

> Hi Alexey,
>
> Did you find the patch is looking good and is ready to be merged?
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 15 мар. 2018 г. в 11:19, Alexey Kukushkin :
>
> > Just found the fix is ready - I will review it today or tomorrow.
> >
>

Test 150 clients migrated to `Cache [6]` suite

[Дмитрий Рябов]

Hello Igniters!



I migrated test `IgniteCache150ClientsTest` from `150 Clients` suite to `
Cache [6]`, because `150 Clients` contains only one test class and makes
~6min excess Ignite build.



So, test suites `150 Clients` and `~[Obsolete] 150 Clients` will be deleted
 soon.