Calcite based SQL query engine. Local queries

[Seliverstov Igor]

Hi Igniters,

Working on new generation of Ignite SQL I faced a question: «Do we need local 
queries at all and, if so, what semantic they should have?».

Current planing flow consists of next steps:

1) Parsing SQL to AST
2) Validating AST (against Schema)
3) Optimizing (Building execution graph)
4) Splitting (into query fragments which executes on target nodes)
5) Mapping (query fragments to nodes/partitions)

At last step we check that all Fragment sources (a table or result) have the 
same distribution (in other words all sources have to be co-located)

Planner and Splitter guarantee that all caches in a Fragment are co-located, an 
Exchange is produced otherwise. But if we force local execution we cannot 
produce Exchanges, that means we may face two non-co-located caches inside a 
single query fragment (result of local query planning is a single query 
fragment). So, we cannot pass the check.

Should we throw an exception or omit the check for local query planning or 
prohibit local queries at all?

Your thoughts?

Regards,
Igor

Re: Adding support for Ignite secondary indexes to Apache Calcite planner

[Seliverstov Igor]

Colleagues,

As far as I understand, materialization acts like a special rule, that matches 
some subtree pattern (a leaf part of a query plan) to a star table, which may 
have better cost than the subtree, it replaces. Saying that, in general, there 
is no difference between approaches - they do the same almost in the same way 
but using different API.

My opinion is it’s better to do the deal using rules - it makes overall 
approach consistent.

Regards,
Igor

> 12 дек. 2019 г., в 10:03, Vladimir Ozerov  написал(а):
> 
> Roman,
> 
> What I am trying to understand is what advantage of materialization API you
> see over the normal optimization process? Does it save optimization time,
> or reduce memory footprint, or maybe provide better plans? I am asking
> because I do not see how expressing indexes as materializations fit
> classical optimization process. We discussed Sort <- Scan optimization.
> Let's consider another example:
> 
> LogicalSort[a ASC]
>  LogicalJoin
> 
> Initially, you do not know the implementation of the join, and hence do not
> know it's collation. Then you may execute physical join rules, which
> produce, say, PhysicalMergeJoin[a ASC]. If you execute sort implementation
> rule afterwards, you may easily eliminate the sort, or make it simpler
> (e.g. remove local sorting phase), depending on the distribution. In other
> words, proper implementation of sorting optimization assumes that you have
> a kind of SortRemoveRule anyway, irrespectively of whether you use
> materializations or not, because sorting may be injected on top of any
> operator. With this in mind, the use of materializations doesn't make the
> planner simpler. Neither it improves the outcome of the whole optimization
> process.
> 
> What is left is either lower CPU or RAM usage? Is this the case?
> 
> ср, 11 дек. 2019 г. в 18:37, Roman Kondakov :
> 
>> Vladimir,
>> 
>> the main advantage of the Phoenix approach I can see is the using of
>> Calcite's native materializations API. Calcite has advanced support for
>> materializations [1] and lattices [2]. Since secondary indexes can be
>> considered as materialized views (it's just a sorted representation of
>> the same table) we can seamlessly use views to simulate indexes behavior
>> for Calcite planner.
>> 
>> 
>> [1] https://calcite.apache.org/docs/materialized_views.html
>> [2] https://calcite.apache.org/docs/lattice.html
>> 
>> --
>> Kind Regards
>> Roman Kondakov
>> 
>> 
>> On 11.12.2019 17:11, Vladimir Ozerov wrote:
>>> Roman,
>>> 
>>> What is the advantage of Phoenix approach then? BTW, it looks like
>> Phoenix
>>> integration with Calcite never made it to production, did it?
>>> 
>>> вт, 10 дек. 2019 г. в 19:50, Roman Kondakov >> :
>>> 
 Hi Vladimir,
 
 from what I understand, Drill does not exploit collation of indexes. To
 be precise it does not exploit index collation in "natural" way where,
 say, we a have sorted TableScan and hence we do not create a new Sort.
 Instead of it Drill always create a Sort operator, but if TableScan can
 be replaced with an IndexScan, this Sort operator is removed by the
 dedicated rule.
 
 Lets consider initial an operator tree:
 
 Project
  Sort
TableScan
 
 after applying rule DbScanToIndexScanPrule this tree will be converted
>> to:
 
 Project
  Sort
IndexScan
 
 and finally, after applying DbScanSortRemovalRule we have:
 
 Project
  IndexScan
 
 while for Phoenix approach we would have two equivalent subsets in our
 planner:
 
 Project
  Sort
TableScan
 
 and
 
 Project
  IndexScan
 
 and most likely the last plan  will be chosen as the best one.
 
 --
 Kind Regards
 Roman Kondakov
 
 
 On 10.12.2019 17:19, Vladimir Ozerov wrote:
> Hi Roman,
> 
> Why do you think that Drill-style will not let you exploit collation?
> Collation should be propagated from the index scan in the same way as
>> in
> other sorted operators, such as merge join or streaming aggregate.
 Provided
> that you use converter-hack (or any alternative solution to trigger
 parent
> re-analysis).
> In other words, propagation of collation from Drill-style indexes
>> should
 be
> no different from other sorted operators.
> 
> Regards,
> Vladimir.
> 
> вт, 10 дек. 2019 г. в 16:40, Zhenya Stanilovsky
 > :
> 
>> 
>> Roman just as fast remark, Phoenix builds their approach on
>> already existing monolith HBase architecture, most cases it`s just a
 stub
>> for someone who wants use secondary indexes with a base with no
>> native support of it. Don`t think it`s good idea here.
>> 
>>> 
>>> 
>>> --- Forwarded message ---
>>> From: "Roman Kondakov" < kondako...@mail.ru.invalid >
>>> To:  dev@ignite.apache.org
>>> Cc:
>>> Subject: Addin

Re: SQL dialects supported by Calcite

[Seliverstov Igor]

Denis,

> Is it true that Calcite can parse MySQL, Oracle

Yes, it can.

> Do I need to select a dialect globally or can it be set on a per-query
> level?

Technically a parser, a validator, a planner, other components are created and 
configured for each query call (because they are stateful). So you may 
configure it per query, or hardcode a desired dialect, or get it from 
configuration - it’s up to you, but we expect parser configuration to be (more 
or less) static, it is a part of initial framework configuration. I think we 
should not allow such parameters (as a dialect) changing.

Regards,
Igor

> 27 дек. 2019 г., в 07:47, Denis Magda  написал(а):
> 
> Igor S., Roman and others who involved in Calcite prototyping,
> 
> Is it true that Calcite can parse MySQL, Oracle, ANSI-99 and other dialects
> as suggested by this page?
> https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html
> 
> Do I need to select a dialect globally or can it be set on a per-query
> level?
> 
> -
> Denis

Re: SQL dialects supported by Calcite

[Seliverstov Igor]

I guess you are thinking about transparent migration from Oracle for
example.

>From user perspective it's really cool, but we will be forced to maintain
all these dialects and fully test them. Also I heard about several
inconsistencies between how it should and how it actually works. All these
issues become ours if we declare several dialects support.

So, I think multi dialects support will bring more troubles than benefits.

Regards,
Igor

сб, 28 дек. 2019 г., 0:42 Denis Magda :

> Igor,
>
> When you are saying that we should not allow the dialect changing, are you
> referring to changes in runtime when a node is already up-and-running?
> Generally, it will be more than enough if a dialect can be set statically
> and globally -- the user selects the dialect for the entire cluster via a
> configuration parameter and starts the nodes after that.
>
> -
> Denis
>
>
> On Fri, Dec 27, 2019 at 7:05 AM Seliverstov Igor 
> wrote:
>
> > Denis,
> >
> > > Is it true that Calcite can parse MySQL, Oracle
> >
> > Yes, it can.
> >
> > > Do I need to select a dialect globally or can it be set on a per-query
> > > level?
> >
> > Technically a parser, a validator, a planner, other components are
> created
> > and configured for each query call (because they are stateful). So you
> may
> > configure it per query, or hardcode a desired dialect, or get it from
> > configuration - it’s up to you, but we expect parser configuration to be
> > (more or less) static, it is a part of initial framework configuration. I
> > think we should not allow such parameters (as a dialect) changing.
> >
> > Regards,
> > Igor
> >
> > > 27 дек. 2019 г., в 07:47, Denis Magda  написал(а):
> > >
> > > Igor S., Roman and others who involved in Calcite prototyping,
> > >
> > > Is it true that Calcite can parse MySQL, Oracle, ANSI-99 and other
> > dialects
> > > as suggested by this page?
> > >
> >
> https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html
> > >
> > > Do I need to select a dialect globally or can it be set on a per-query
> > > level?
> > >
> > > -
> > > Denis
> >
> >
>

Re: SQL dialects supported by Calcite

[Seliverstov Igor]

Forgot to mention, it's better to set chosen dialect per session, for
example by a command

"alter session sed dialect=ORACLE"

Or a part of connection url.

Global configuration parameter isn't a good choice.

сб, 28 дек. 2019 г., 11:02 Seliverstov Igor :

> I guess you are thinking about transparent migration from Oracle for
> example.
>
> From user perspective it's really cool, but we will be forced to maintain
> all these dialects and fully test them. Also I heard about several
> inconsistencies between how it should and how it actually works. All these
> issues become ours if we declare several dialects support.
>
> So, I think multi dialects support will bring more troubles than benefits.
>
> Regards,
> Igor
>
> сб, 28 дек. 2019 г., 0:42 Denis Magda :
>
>> Igor,
>>
>> When you are saying that we should not allow the dialect changing, are you
>> referring to changes in runtime when a node is already up-and-running?
>> Generally, it will be more than enough if a dialect can be set statically
>> and globally -- the user selects the dialect for the entire cluster via a
>> configuration parameter and starts the nodes after that.
>>
>> -
>> Denis
>>
>>
>> On Fri, Dec 27, 2019 at 7:05 AM Seliverstov Igor 
>> wrote:
>>
>> > Denis,
>> >
>> > > Is it true that Calcite can parse MySQL, Oracle
>> >
>> > Yes, it can.
>> >
>> > > Do I need to select a dialect globally or can it be set on a per-query
>> > > level?
>> >
>> > Technically a parser, a validator, a planner, other components are
>> created
>> > and configured for each query call (because they are stateful). So you
>> may
>> > configure it per query, or hardcode a desired dialect, or get it from
>> > configuration - it’s up to you, but we expect parser configuration to be
>> > (more or less) static, it is a part of initial framework configuration.
>> I
>> > think we should not allow such parameters (as a dialect) changing.
>> >
>> > Regards,
>> > Igor
>> >
>> > > 27 дек. 2019 г., в 07:47, Denis Magda  написал(а):
>> > >
>> > > Igor S., Roman and others who involved in Calcite prototyping,
>> > >
>> > > Is it true that Calcite can parse MySQL, Oracle, ANSI-99 and other
>> > dialects
>> > > as suggested by this page?
>> > >
>> >
>> https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html
>> > >
>> > > Do I need to select a dialect globally or can it be set on a per-query
>> > > level?
>> > >
>> > > -
>> > > Denis
>> >
>> >
>>
>

New SQL engine, review/feedback request

[Seliverstov Igor]

Hi guys,

Some of you know, I'm working on a brand new Calcite based SQL engine.

So, now I have some intermediate results and need a feedback from community.

A feature branch with a concept of a module is ignite-12248 
. The module is going to be 
continuously enhanced with new functionality.

I’ll appreciate If you share your thoughts.

Additionally a next patch required a review:

https://issues.apache.org/jira/browse/IGNITE-12448 

https://github.com/apache/ignite/pull/7272 


There are quite a lot of changes because I'm still trying to find a best way of 
how to organize the engine parts cross dependencies and code in general.

Regards,
Igor

Re: [IGNITE-12582] Configuration by property

[Seliverstov Igor]

Isn't a better way just to add a new annotation?

Like RepositoryConfig but from v2 package for example.

This case whose who already use it won't suffer.

Also it's would be great to provide a way to escape constants which are
similar to spel expressions.

Regards,
Igor

чт, 30 янв. 2020 г., 13:18 Sergey Chernolyas :

> Hi igniters!
>
> Presently, Spring Data for Ignite can't be configured dynamically. I mean ,
> than I defines repository by the code:
>
> @Repository
> @RepositoryConfig(cacheName = "Calendar")
> public interface CalendarRepository extends IgniteRepository String> {
> List findByName(String name);
> }
>
> But I need to configure used cache dynamically ( at runtime). To solve
> this problem is proposed to use Spring Expression Language. By the
> way, I will have possibility to use the code:
>
> @Repository
> @RepositoryConfig(cacheName = "${cache.calendar.name}")
> public interface CalendarRepository extends IgniteRepository String> {
> List findByName(String name);
> }
>
> And property "cache.calendar.name" can be configured as usual property
> at Spring Framework.
>
> But way brakes current configuration way and I would ask about how we
> can set cache name by  expression or constant string.
>
> I see options:
>
> 1) field "cacheName" will be able to process expression and constant
> string. The code of repository factory will analyse the field.
>
> 2) create new field for expressions
>
>
> What is best way?
>
>
> --
> -
>
> With best regards, Sergey Chernolyas
>

Re: Mark MVCC with @IgniteExperimental

[Seliverstov Igor]

Note that someone uses it

Main problem is a recovery process when persistence enabled and a cluster have 
more than one node. 

It is a problem even for regular transactional caches, the main difference - 
MVCC detects any inconsistencies while regular transactional caches may ignore 
it without any notification

In other cases it works fine and provides promised guaranties.

Of course there are several minor issues like performance ones, but there is a 
plan how to solve them (I could share it if anybody is curious)

My opinion we should solve consistency issues first and finalize MVCC after 
that.

Until that it’s OK to have it as experimental feature.

Regards,
Igor

> 6 февр. 2020 г., в 21:25, Alexei Scherbakov  
> написал(а):
> 
> I'm strongly support removal of MVCC from master.
> 
> At the current state it bloats code base and should be reworked from
> scratch using separate code base.
> 
> чт, 6 февр. 2020 г. в 19:45, Ilya Kasnacheev :
> 
>> Hello!
>> 
>> Please keep in mind that you need to create a separate proposal voting
>> thread if you really like it to count. I wonder if Dmitry Pavlov can help
>> us with the procedure.
>> 
>> Otherwise, I think it makes total sense to restrict MVCC clusters to only
>> have MVCC caches or REPLICATED TRANSACTIONAL caches (are they compatible in
>> our current implementation) and no ATOMIC caches at all.
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> чт, 6 февр. 2020 г. в 19:41, Maxim Muzafarov :
>> 
>>> Ilya,
>>> 
>>> 
>>> 1. MVCC support requires code maintenance for other developed features
>>> even if has not used and disabled. Currently, we've got an x2 level of
>>> difficulty for implementation of new features.
>>> 
>>> 2. It would be much easy to develop and support clusters with
>>> mvcc-caches only rather than have a mixed configuration. With this
>>> option we can dramatically reduce the amount of codebase removing from
>>> mvcc-branch local, atomic, tx caches.
>>> 
>>> 
>>> So, I'm +1 to remove it from the master branch and mark the current
>>> API with @IgniteExperimental.
>>> 
>>> On Thu, 6 Feb 2020 at 19:29, Ilya Kasnacheev 
>>> wrote:
 
 Hello!
 
 Why would we drop MVCC!?
 
 I can totally imagine a scenario where a large Ignite user surfaces
>> with
 fixes for MVCC mode, if it is kept as an experimental feature. Then
>> maybe
 it will graduate to beta some time in future.
 
 If it does too much strain on the TC, let's discuss that, but I don't
 remember problems with TC load lately, so maybe this is a moot point.
 
 Regards,
 --
 Ilya Kasnacheev
 
 
 чт, 6 февр. 2020 г. в 15:27, Nikolay Izhikov :
 
>> By the way, is there any reason to have this code in the master
>>> branch
> 
> I support removal MVCC from master.
> 
> 
>> 6 февр. 2020 г., в 15:26, Andrey Gura 
>> написал(а):
>> 
>> +1
>> 
>> By the way, is there any reason to have this code in the master
>> branch? It seems feature is in unsupportable state and just wastes
>> TC
>> time and our effort to support MVCC related tests.
>> 
>> On Thu, Feb 6, 2020 at 2:44 PM Alexey Goncharuk
>>  wrote:
>>> 
>>> Agree, let's mark MVCC experimental.
>>> 
>>> Stephen, the annotation serves as an additional
>> documentation-style
> marker.
>>> For now there are no compile-time warnings when the API is used.
>>> 
>>> чт, 6 февр. 2020 г. в 14:35, Stephen Darlington <
>>> stephen.darling...@gridgain.com>:
>>> 
 Yes! I’ve already seen people try to use this without awareness
>>> that
> it’s
 not production ready.
 
 What happens with the annotation, incidentally? Is it just in the
 documentation or do you get a compile-time warning?
 
> On 6 Feb 2020, at 11:32, Nikolay Izhikov 
>>> wrote:
> 
> Hello, Igniters.
> 
> Should we mark MVCC feature with the new @IgniteExperimental?
> 
> We explicitly note users that MVCC has beta status, for now [1]
> 
>> Beta version of Transactional SQL and MVCC
>> In Ignite v2.7, Transactional SQL and MVCC are released as beta
 versions to allow users to experiment and share feedback.
>> This version of Transactional SQL and MVCC should not be
>>> considered
> for
 production.
> 
> 
> [1]
> https://apacheignite.readme.io/docs/multiversion-concurrency-control
> 
> 
 
 
 
> 
> 
>>> 
>> 
> 
> 
> -- 
> 
> Best regards,
> Alexei Scherbakov

Workaround for possible NPE in Selector.open()

[Seliverstov Igor]

Hi Igniters,

Today I noticed quite old code aimed to workaround an old bug in 1.5 JDK 
revision which was fixed in rev 1.7

org/apache/ignite/internal/util/nio/GridNioServer.java:161

Since we widely use lambdas now (so Ignite requires at least Java 1.8) do we 
really need it now?

I would like to remove it.

Regards,
Igor

Re: Suggestion to improve deadlock detection

[Seliverstov Igor]

Ivan,

We have to design configuration carefully.
There are possible issues with compatibility and this API is public, it
might be difficult to redesign it after while.

Since there is a ticket about MVCC related configuration parts [1] (TxLog
memory region/size, vacuum thread count and intervals, etc)
I suggest solving the issue in scope of that ticket.

At now, let's introduce a couple of system variables for an ability to
change the behavior while development phase.

[1] https://issues.apache.org/jira/browse/IGNITE-7966

Regards,
Igor

чт, 20 дек. 2018 г. в 16:29, Павлухин Иван :

> Hi,
>
> I prepared a patch with deadlock detection algorithm implementation
> (you can find it from ticket [1]).
>
> Now I would like to discuss options for configuring deadlock
> detection. From a usability standpoint following need to be supported:
> 1. Turn deadlock detection on/off (on by default).
> 2. Configure a delay between starting waiting for a lock and sending
> first deadlock detection message (e.g. 1 second by default).
> 3. Something else?
>
> I can suggest following options:
> 1. Use a single long for configuration. Values < 0 means disabled
> deadlock detection. 0 means starting detection without a delay. Values
> > 0 specifies a delay in milliseconds before starting detection.
> 2. Use a boolean flag for turning detection on/off. Use a long value
> for configuring delay.
>
> Also there are number of ways to pass this values to Ignite. I do no
> have much experience in adding new configuration options to Ignite and
> I need an advice here. For example, it could be done by adding
> additional properties to TransactionConfiguration class. Another way
> is using java system properties.
>
> Igor, Vladimir and others what do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9322
> ср, 19 дек. 2018 г. в 15:51, Павлухин Иван :
> >
> > Igor,
> >
> > I see your points. And I agree to start with "lightweight
> > implementation". Today we have 2PL and there is no activity on
> > implementing rollback to savepoint. And if we do it in the future we
> > will have to return to the subject of deadlock detection anyway.
> >
> > I will proceed with "forward-only" approach.
> > вт, 18 дек. 2018 г. в 18:33, Seliverstov Igor :
> > >
> > > Ivan,
> > >
> > > I would prefer forward-only implementation even knowing it allows false
> > > positive check results.
> > >
> > > Why I think so:
> > >
> > > 1) From my experience, when we have any future is waiting for reply, we
> > > have to take a failover into consideration.
> > > Usually failover implementations are way more complex than an initial
> > > algorithm itself.
> > > Forward-only version doesn't need any failover implementation as it
> expects
> > > failed probes (the probes didn't face a deadlock ).
> > >
> > > 2) Simple lightweight feature implementation is a really good point to
> > > start - it may be extended if needed but really often such extending
> > > doesn't need at all.
> > >
> > > 3) Any implementation allow false positive result - we are working with
> > > distributed system, there are a bunch of processes happens at the same
> time
> > > and,
> > > for example, concurrently happened rollback on timeout may solve a
> deadlock
> > > but probe is finished at this time, so- there is a rollback on deadlock
> > > also as a result.
> > >
> > > 4) The described case (when false positive result is probable) isn't
> usual
> > > but, potentially, extremely rare, I don't think we should solve it
> since it
> > > doesn't break the grid.
> > >
> > > Regards,
> > > Igor
> > >
> > > вт, 18 дек. 2018 г. в 17:57, Павлухин Иван :
> > >
> > > > Hi folks,
> > > >
> > > > During implementation of edge-chasing deadlock detection algorithm in
> > > > scope of [1] it has been realized that basically we have 2 options
> for
> > > > "chasing" strategy. I will use terms Near when GridNearTxLocal is
> > > > assumed and Dht when GridDhtTxLocal (tx which updates primary
> > > > partition). So, here is 2 mentioned strategies:
> > > > 1. Send initial probe when Dht tx blocks waiting for another tx to
> > > > release a key. Initial probe is sent from waiting Dht tx to Near tx
> > > > holding a lock. If receiving Near tx is blocked as well then it
> relays
> > > > the probe to Dht tx it awaits response f

Re: [DISCUSSION][IEP-35] Metrics configuration

[Seliverstov Igor]

Igniters,

One more question on topic.

Should we preserve metrics configuration on restart? (I think we should)

If so, which configuration use after restart? Defined in config file or saved 
in config storage? (I guess, saved configuration should have a priority)

So, how to tell users that any changes in configuration file have no effect on 
Ignite configuration after first start?

I think there are too many open questions and (at least at now) we should 
provide only JMX API until all of the questions are clarified.

Regards,
Igor

> 4 июля 2019 г., в 19:55, Nikolay Izhikov  написал(а):
> 
> Hello, Andrey.
> 
>> 3. I can't imagine that adequate values will be chosen on project
>> setup stage.
> 
> Configuration file required in the case we adds new node or replace existing 
> to the cluster.
> Use can have parameters similar to Ignite configuration, log configuration 
> files.
> 
>> My proposal is adding API for boundaries configuration to the metrics
>> framework and expose it via JMX
> 
> Agree. I think we should have both:
> 
> 1. Configuration file.
> 2. JMX API to change bounaries of histogram *and HitRateMetric params*.
> 
> But, if you and other community member are against config file, let's have 
> only JMX.
> Seems, JMX will provide required level of configurability for metrics.
> 
> 
> В Чт, 04/07/2019 в 17:53 +0300, Andrey Gura пишет:
>> Igniters,
>> 
>> I rethought the issue and I see some problems:
>> 
>> 1. It seems that in most cases bucket boundaries configuration will be
>> problem for user. Absolute values for latency boundaries it is very
>> odd choice.
>> 2. Also seems that latency for most caches (if we configure cache
>> metrics fro example) will be similar.
>> 3. I can't imagine that adequate values will be chosen on project
>> setup stage. So chosen values should be changed in the future.
>> 
>> Solution with configuration file looks unnatural and creates more
>> problems than could solve.
>> 
>> My proposal is adding API for boundaries configuration to the metrics
>> framework and expose it via JMX (at this step). It still provides
>> configuration possibility but don't force user to do it.
>> 
>> Also we should chose default values for bucket boundaries. And it is
>> most complex problem at the moment :) Let's discuss it.
>> 
>> 
>> 
>> 
>> 
>> On Wed, Jul 3, 2019 at 4:49 PM Andrey Gura  wrote:
>>> 
>>> Nikolai,
>>> 
>>> Metric is disabled if it doesn't allocate any memory and doesn't
>>> update any variable because doesn't have any value. Ideally disabling
>>> metrics for some cache should be equal to cache stopping.
>>> 
>>> On Fri, Jun 28, 2019 at 1:02 PM Nikolay Izhikov  wrote:
 
 Hello, Alexey.
 
 Thanks for the feedback!
 
> My only concert is that we should have the metrics framework configuration
> as the first-citizen of the framework itself
 
 Yes. I planned to add `void configure(String param)` method to the metric 
 API.
 
> but change the metrics parameters in
> runtime from JMX or command-line, etc.
 
 I've add requirement of JMX method to the ticket:
 
 https://issues.apache.org/jira/browse/IGNITE-11927
 
> Another concern is to have an
> ability to disable/enable metrics per metrics group/prefix.
 
 Yes, we discusss it.
 But, let's make it clear:
 
 *What is disabling metric?*
 
 Looks like exporter filter solve this task.
 
 В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет:
> Nikolay,
> 
> My only concert is that we should have the metrics framework configuration
> as the first-citizen of the framework itself. This way, we can configure
> the metrics not only from file, but change the metrics parameters in
> runtime from JMX or command-line, etc. Another concern is to have an
> ability to disable/enable metrics per metrics group/prefix.
> 
> The logger-like configuration meets these suggestions given that the
> configuration is generalized into the metrics framework.
> 
> What do you think?
> 
> чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov :
> 
>> Hello, Igniters.
>> 
>> As you may know, I've contributed Phase1 [1] for IEP-35 [2].
>> Now we have metrics subsystem and can create and export any metrics from
>> Ignite.
>> 
>> I think user(administrator of Ignite) should be able to configure some
>> metrics params in a common way [3]
>> 
>> I propose to use the same way from logging frameworks.
>> We should define some file format Ignite can understand.
>> An administrator fills configuration file to configure one or several
>> metrics.
>> Ignite will analyze the file and use provided params during metrics
>> creation.
>> 
>> For now, we have 2 types of metrics that should be configured:
>> 
>>*   HistrogramMetric [4]
>>This metric is a count of measurement that falls into
>

Re: New SQL execution engine

[Seliverstov Igor]

Nikolay,

The main issue - there is no *selection*.

There is a field of knowledge - relational algebra, which describes how to 
transform relational expressions saving their semantics, and a couple of 
implementations (Calcite is only one written in Java).

There are only two alternatives:

1) Implementing white papers from scratch
2) Adopting Calcite to our needs.

The second way was chosen by several other projects, there is experience, there 
is a list of known issues (like using indexes) so, almost everything is already 
done for us.

Implementing a planner is a big deal, I think anybody understands it there. 
That's why our proposal to reuse others experience is obvious.

If you have an alternative - you're welcome, I'll gratefully listen to you.

The main question isn't "WHAT" but "HOW" - that's the discussion topic from my 
point of view.

Regards,
Igor

> 27 сент. 2019 г., в 16:37, Nikolay Izhikov  написал(а):
> 
> Roman.
> 
>> Nikolay, Maxim, I understand that our arguments may not be as obvious 
>> for you as it obvious for SQL team. So, please arrange your questions in 
>> a more constructive way.
> 
> What is SQL team?
> I only know Ignite community :)
> 
> Please, share you knowledge in IEP.
> I want to join to the process of engine *selection*.
> It should start with the requirements to such engine.
> Can you write it in IEP, please?
> 
> My point is very simple:
> 
> 1. We made the wrong decision with H2
> 2. We should make a well-thought decision about the new engine.
> 
>> How many tickets would satisfy you?
> 
> You write about "issueS" with the H2.
> All I see is one open ticket.
> IEP doesn't provide enough information.
> So it's not about the number of tickets, it's about
> 
>> These two points (single map-reduce execution and inflexible optimizer) 
>> are the main problems with the current engine.
> 
> We may come to the point when Calcite(or any other engine) brings us third 
> and other "main problems".
> This is how it happens with H2.
> 
> Let's start from what we want to get with the engine and move forward from 
> this base.
> What do you think?
> 
> 
> 
> В Пт, 27/09/2019 в 16:15 +0300, Roman Kondakov пишет:
>> Maxim, Nikolay,
>> 
>> I've listed two issues which show the ideological flaws of the current 
>> engine.
>> 
>> 1. IGNITE-11448 - Open. This ticket describes the impossibility of 
>> executing queries which can not be fit in the hardcoded one pass 
>> map-reduce paradigm.
>> 
>> 2. IGNITE-6085 - Closed (won't fix) - This ticket describes the second 
>> major problem with the current engine: H2 query optimizer is very 
>> primitive and can not perform many useful optimizations.
>> 
>> These two points (single map-reduce execution and inflexible optimizer) 
>> are the main problems with the current engine. It means that our engine 
>> is currently  suitable for execution only a very limited subset of the 
>> typical SQL queries. For example it can not even run most of the TPC-H 
>> benchmark queries because they don't fit to the simple map-reduce paradigm.
>> 
>>> All I see is links to two tickets:
>> 
>> How many tickets would satisfy you? I named two. And it looks like it is 
>> not enough from your point of view. Ok, so how many is enough? The set 
>> of problems caused by listed above tickets is infinite, therefore I can 
>> not create a ticket for each of them.
>>> Tech details also should be added.
>> 
>> Tech details are in the tickets.
>> 
>>> We can't discuss such a huge change as an execution engine replacement with 
>>> descrition like:
>>> "No data co-location control, i.e. arbitrary data can be returned silently" 
>>> or
>>> "Low control on how query executes internally, as a result we have limited 
>>> possibility to implement improvements/fixes."
>> 
>> Why not? Don't you understand these problems? Or you don't think this is 
>> a problem?
>> 
>>> Let's make these descriptions more specific.
>> 
>> What do you mean by "more specific"? What is the criteria of the 
>> specific description?
>> 
>> 
>> 
>> Nikolay, Maxim, I understand that our arguments may not be as obvious 
>> for you as it obvious for SQL team. So, please arrange your questions in 
>> a more constructive way.
>> 
>> Thank you!

Re: New SQL execution engine

[Seliverstov Igor]

Nikolay,

At last we have better questions.

There is no decision, here we should decide.

Doing nothing isn’t a decision, it’s just doing nothing

Spark Catalyst is a good example, but under the hood it has absolutely the same 
idea, but adopted to Spark. Calcite is the same, but general. That’s why it’s 
better start point.

Implementing an engine from scratch is really cool, but looks like inventing a 
bicycle, don’t think it makes sense. At least I against this option.

I added requirements to IEP (as you asked), you may see it’s in DRAFT state and 
will be complemented by details.

We have some thoughts on how to make smooth replacement, but at first we should 
decide what to replace and what with.

At now Calcite based engine is placed in different module, we checked it can 
build execution graph for both local and distributed cases, it has good 
expandability. 
We talked to Calcite community to identify possible future issues and 
everything points to the fact it’s the best option. 
It’s possible to develop it as an experimental extension at first (not a 
replacement) until we make sure that it works as expected. This way there are 
no risks for anybody who uses Ignite on production environment.

Regards,
Igor


> 27 сент. 2019 г., в 17:25, Nikolay Izhikov  написал(а):
> 
> Igor.
> 
>> The main issue - there is no *selection*.
> 
> 1. I don't remember community decision about this.
> 
> 2. We should avoid to make such long-term decision so quickly.
> We done this kind of decision with H2 and come to the point when we should 
> review it.
> 
>> 1) Implementing white papers from scratch
>> 2) Adopting Calcite to our needs.
> 
> The third option don't fix issues we have with H2.
> The fourth option I know is using spark-catalyst.
> 
> What is wrong with writing engine from scratch?
> 
> I ask you to start with engine requirements.
> Can we, please, discuss it?
> 
>> If you have an alternative - you're welcome, I'll gratefully listen to you.
> 
> We have alternative for now - H2 based engine.
> 
>> The main question isn't "WHAT" but "HOW" - that's the discussion topic from 
>> my point of view.
> 
> When we make a decision about engine we can discuss roadmap for replacement.
> One more time - replacement of SQL engine to some more customizable make 
> sense for me.
> But, this kind of decisions need carefull discussion.
> 
> В Пт, 27/09/2019 в 17:08 +0300, Seliverstov Igor пишет:
>> Nikolay,
>> 
>> The main issue - there is no *selection*.
>> 
>> There is a field of knowledge - relational algebra, which describes how to 
>> transform relational expressions saving their semantics, and a couple of 
>> implementations (Calcite is only one written in Java).
>> 
>> There are only two alternatives:
>> 
>> 1) Implementing white papers from scratch
>> 2) Adopting Calcite to our needs.
>> 
>> The second way was chosen by several other projects, there is experience, 
>> there is a list of known issues (like using indexes) so, almost everything 
>> is already done for us.
>> 
>> Implementing a planner is a big deal, I think anybody understands it there. 
>> That's why our proposal to reuse others experience is obvious.
>> 
>> If you have an alternative - you're welcome, I'll gratefully listen to you.
>> 
>> The main question isn't "WHAT" but "HOW" - that's the discussion topic from 
>> my point of view.
>> 
>> Regards,
>> Igor
>> 
>>> 27 сент. 2019 г., в 16:37, Nikolay Izhikov  написал(а):
>>> 
>>> Roman.
>>> 
>>>> Nikolay, Maxim, I understand that our arguments may not be as obvious 
>>>> for you as it obvious for SQL team. So, please arrange your questions in 
>>>> a more constructive way.
>>> 
>>> What is SQL team?
>>> I only know Ignite community :)
>>> 
>>> Please, share you knowledge in IEP.
>>> I want to join to the process of engine *selection*.
>>> It should start with the requirements to such engine.
>>> Can you write it in IEP, please?
>>> 
>>> My point is very simple:
>>> 
>>> 1. We made the wrong decision with H2
>>> 2. We should make a well-thought decision about the new engine.
>>> 
>>>> How many tickets would satisfy you?
>>> 
>>> You write about "issueS" with the H2.
>>> All I see is one open ticket.
>>> IEP doesn't provide enough information.
>>> So it's not about the number of tickets, it's about
>>> 
>>>> These two p

Re: New SQL execution engine

[Seliverstov Igor]

Nikolay,

> What project hosted Calcite based engine?


Currently the prototype is placed in my personal Ignite fork. I need an 
appropriate ticket before pushing it to ASF git repository. 
At first, I think, we should discuss the idea in general.

> Personally, I'm against the support of two independent implementation of SQL 
> engine for several releases.


I don’t like the idea to have two engines too. But even development the engine 
on top of Calcite library is still a big deal. 
I not sure it will be ready, no, I sure it WONT be ready by Ignite3 release. So 
I mentioned the option to have two engines at the same time.

> Let's start with the IEP clarification and replace the SQL engine with the 
> best one for Ignite good.

Of course, but anyway it’s good to make familiar with a couple of examples it 
already describes and clarify some additional questions the community may ask.

Regards,
Igor

> 27 сент. 2019 г., в 18:22, Nikolay Izhikov  написал(а):
> 
> Igor.
> 
>> There is no decision, here we should decide.
> 
> Great.
> 
>> At now Calcite based engine is placed in different module
> 
> What project hosted Calcite based engine?
> 
>> It’s possible to develop it as an experimental extension at first (not a 
>> replacement)
> 
> For me, Ignite 3 are the place where the new engine has to be placed.
> Personally, I'm against the support of two independent implementation of SQL 
> engine for several releases.
> 
> Ignite has too many partially implemented features to include on more :)
> 
> Let's start with the IEP clarification and replace the SQL engine with the 
> best one for Ignite good.
> 
> 
> В Пт, 27/09/2019 в 18:08 +0300, Seliverstov Igor пишет:
>> Nikolay,
>> 
>> At last we have better questions.
>> 
>> There is no decision, here we should decide.
>> 
>> Doing nothing isn’t a decision, it’s just doing nothing
>> 
>> Spark Catalyst is a good example, but under the hood it has absolutely the 
>> same idea, but adopted to Spark. Calcite is the same, but general. That’s 
>> why it’s better start point.
>> 
>> Implementing an engine from scratch is really cool, but looks like inventing 
>> a bicycle, don’t think it makes sense. At least I against this option.
>> 
>> I added requirements to IEP (as you asked), you may see it’s in DRAFT state 
>> and will be complemented by details.
>> 
>> We have some thoughts on how to make smooth replacement, but at first we 
>> should decide what to replace and what with.
>> 
>> At now Calcite based engine is placed in different module, we checked it can 
>> build execution graph for both local and distributed cases, it has good 
>> expandability. 
>> We talked to Calcite community to identify possible future issues and 
>> everything points to the fact it’s the best option. 
>> It’s possible to develop it as an experimental extension at first (not a 
>> replacement) until we make sure that it works as expected. This way there 
>> are no risks for anybody who uses Ignite on production environment.
>> 
>> Regards,
>> Igor
>> 
>> 
>>> 27 сент. 2019 г., в 17:25, Nikolay Izhikov  написал(а):
>>> 
>>> Igor.
>>> 
>>>> The main issue - there is no *selection*.
>>> 
>>> 1. I don't remember community decision about this.
>>> 
>>> 2. We should avoid to make such long-term decision so quickly.
>>> We done this kind of decision with H2 and come to the point when we should 
>>> review it.
>>> 
>>>> 1) Implementing white papers from scratch
>>>> 2) Adopting Calcite to our needs.
>>> 
>>> The third option don't fix issues we have with H2.
>>> The fourth option I know is using spark-catalyst.
>>> 
>>> What is wrong with writing engine from scratch?
>>> 
>>> I ask you to start with engine requirements.
>>> Can we, please, discuss it?
>>> 
>>>> If you have an alternative - you're welcome, I'll gratefully listen to you.
>>> 
>>> We have alternative for now - H2 based engine.
>>> 
>>>> The main question isn't "WHAT" but "HOW" - that's the discussion topic 
>>>> from my point of view.
>>> 
>>> When we make a decision about engine we can discuss roadmap for replacement.
>>> One more time - replacement of SQL engine to some more customizable make 
>>> sense for me.
>>> But, this kind of decisions need carefull discussion.
>>> 
>>> В Пт, 27/09/2019 в 17:08 +0300, Seliver

Re: New SQL execution engine

[Seliverstov Igor]

Nikolay,

The document you edited is wrong (and outdated).

Since the author meant another idea, I decided not to change IEP-35 and
create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
It's already have a number of key requirements.

Regards,
Igor

вт, 1 окт. 2019 г., 6:14 Nikolay Izhikov :

> Hello, Igniters.
>
> I extends IEP [1] with the tickets caused by H2 limitations.
>
> Please, let's write down requirements for engine in the IEP.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
>
> В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> > Ivan, we need more of these discussions, totally agree with you ;)
> >
> > I've updated the Motivation paragraph outlining some high-level users we
> > see by working with our users. Hope it helps. Let's carry on and let me
> > send a note to Apache Calcite community.
> >
> > -
> > Denis
> >
> >
> > On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin 
> wrote:
> >
> > > Folks,
> > >
> > > Thanks everyone for a hot discussion! Not every open source community
> > > has such open and boiling discussions. It means that people here
> > > really do care. And I am proud of it!
> > >
> > > As I understood, nobody is strictly against the proposed initiative.
> > > And I am glad that we can move forward (with some steps back along the
> > > way).
> > >
> > > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov :
> > > >
> > > > Hello, Denis.
> > > >
> > > > Thanks for the clarifications.
> > > >
> > > > Sounds good for me.
> > > > All I try to say in this thread:
> > > > Guys, please, let's take a step back and write down
> requirements(what we
> > >
> > > want to get with SQL engine).
> > > > Which features and use-cases are primary for us.
> > > >
> > > > I'm sure you have done it, already during your research.
> > > >
> > > > Please, share it with the community.
> > > >
> > > > I'm pretty sure we would back to this document again and again during
> > >
> > > migration.
> > > > So good written design is worth it.
> > > >
> > > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > > Ignite mates, let me try to move the discussion in a constructive
> way.
> > >
> > > It
> > > > > looks like we set a wrong context from the very beginning.
> > > > >
> > > > > Before proposing this idea to the community, some of us were
> > > > > discussing/researching the topic in different groups (the one need
> to
> > >
> > > think
> > > > > it through first before even suggesting to consider changes of this
> > > > > magnitude). The day has come to share this idea with the whole
> > >
> > > community
> > > > > and outline the next actions. But (!) nobody is 100% sure that
> that's
> > >
> > > the
> > > > > right decision. Thus, this will be an *experiment*, some of our
> > >
> > > community
> > > > > members will be developing a *prototype* and only based on the
> > >
> > > prototype
> > > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> > >
> > > hope
> > > > > that nothing has changed and we're on the same page here.
> > > > >
> > > > > Many technical and architectural reasons that justify this project
> have
> > > > > been shared but let me throw in my perspective. There is nothing
> wrong
> > >
> > > with
> > > > > H2, that was the right choice for that time.  Thanks to H2 and
> Ignite
> > >
> > > SQL
> > > > > APIs, our project is used across hundreds of deployments who are
> > > > > accelerating relational databases or use Ignite as a system of
> records.
> > > > > However, these days many more companies are migrating to
> *distributed*
> > > > > databases that speak SQL. For instance, if a couple of years ago 1
> out
> > >
> > > of
> > > > > 10 use cases needed support for multi-joins queries or queries with
> > > > > subselects or efficient memory usage then today there are 5 out of
> 10
> > >
> > > use
> > > > > cases of this kind; in the foreseeable future, it will be a 10 out
> of
> > >
> > > 10.
> > > > > So, the evolution is in progress -- the relational world goes
> > >
> > > distributed,
> > > > > it became exhaustive for both Ignite SQL maintainers and experts
> who
> > >
> > > help
> > > > > to tune it for production usage to keep pace with the evolution
> mostly
> > >
> > > due
> > > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> > >
> > > ready to
> > > > > face the future reality.
> > > > >
> > > > > Luckily, we don't need to rush and don't have the right to rush
> because
> > > > > hundreds existing users have already trusted their production
> > >
> > > environments
> > > > > to Ignite SQL and we need to roll out changes with such a big
> impact
> > > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped
> in
> > >
> > > and
> > > > > agreed to be the first contributors who will be *experimenting*
> with
> > >
> > > the
> > > > > new SQL engine. Let's support them; let's connect them with Apache
> > >
> > > Calcite
> > > > > communit

Re: Ignite community is building Calcite-based prototype

[Seliverstov Igor]

Guys,

The better link: 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
 


Almost everything you may see by the link is the same as Drill guys already 
did, the difference is in details but the idea is the same.

Of course we’ll face many issues while development and I'll appreciate if some 
of you assist us.

Regards,
Igor

> 1 окт. 2019 г., в 12:32, Julian Feinauer  
> написал(а):
> 
> Hi Denis,
> 
> Nice to hear from you and the ignite team... that sounds like an excellent 
> idea. I liked the idea of Ignite since I heard about it (I think when it 
> became TLP back then). So I would be happy to help you if you have specific 
> questions... I‘m currently working on a related topic, namely integrate 
> calcite as SQL Layer into Apache IoTDB .
> 
> Best
> Julian
> 
> Holen Sie sich Outlook für iOS
> 
> Von: Denis Magda 
> Gesendet: Tuesday, October 1, 2019 2:37:20 AM
> An: d...@calcite.apache.org ; dev 
> 
> Betreff: Ignite community is building Calcite-based prototype
> 
> Hey ASF-mates,
> 
> Just wanted to send a note for Ignite dev community who has started
> prototyping
> 
> with a new Ignite SQL engine and Calcite was selected as the most favorable
> option.
> 
> We will truly appreciate if you help us with questions that might hit your
> dev list. Ignite folks have already studied Calcite well enough and carried
> on with the integration, but there might be tricky parts that would require
> your expertise.
> 
> Btw, if anybody is interested in Ignite (memory-centric database and
> compute platform) or would like to learn more details about the prototype
> or join its development, please check these links or send us a note:
> 
>   - https://ignite.apache.org
>   -
>   
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> 
> 
> -
> Denis,
> Ignite PMC Chair

Hello

[Seliverstov Igor]

Hello everyone!

My name is Igor, I'm a novice in this community but wish to put my efforts
into the project)

Could anybody describe me how to join the Ignate project on Apache Ignite
JIRA ?

Thanks in advance
Igor

JIRA ticket

[Seliverstov Igor]

Hi guys!

Doesn't anybody mind if I take IGNITE-4681 ticket?

Regards,
Igor

Re: [VOTE] Apache Ignite 1.9.0 RC2

[Seliverstov Igor]

+1

2 мар. 2017 г. 11:45 ПП пользователь "Valentin Kulichenko" <
valentin.kuliche...@gmail.com> написал:

> +1 (binding)
>
> -Val
>
> On Thu, Mar 2, 2017 at 11:33 AM, Denis Magda  wrote:
>
> > +1 (binding)
> >
> > —
> > Denis
> >
> > > On Mar 2, 2017, at 9:02 AM, Anton Vinogradov  >
> > wrote:
> > >
> > > Dear Sirs!
> > >
> > > We have uploaded the 1.9.0 release candidate to
> > > https://dist.apache.org/repos/dist/dev/ignite/1.9.0-rc2/
> > >
> > > Git tag name is
> > > 1.9.0-rc2
> > >
> > > This release includes the following changes:
> > >
> > > Ignite:
> > > * Added Data streamer mode for DML
> > > * Added Discovery SPI Implementation for Ignite Kubernetes Pods
> > > * SQL: Query can utilize multiple threads
> > > * SQL: Improved distributed SQL support
> > > * Benchmarking simplified and automated
> > > * Fixed licenses generation during build
> > > * ignite-spark module upgraded to Spark 2.0
> > >
> > > Ignite.NET:
> > > * DML support
> > > * TransactionScope API for Ignite transactions support
> > >
> > > Ignite CPP:
> > > * DML support
> > > * Implemented LoadCache
> > > * ContinuousQuery support
> > >
> > > Complete list of closed issues:
> > > https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20IGNITE%20AND%
> > 20fixVersion%20%3D%201.9%20AND%20(status%20%3D%
> > 20closed%20or%20status%20%3D%20resolved)
> > >
> > > DEVNOTES
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=blob_
> > plain;f=DEVNOTES.txt;hb=refs/tags/1.9.0-rc2
> > >
> > > RELEASENOTES
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=blob_
> > plain;f=RELEASE_NOTES.txt;hb=refs/tags/1.9.0-rc2
> > >
> > > Please start voting.
> > >
> > > +1 - to accept Apache Ignite 1.9.0-rc2
> > > 0 - don't care either way
> > > -1 - DO NOT accept Apache Ignite 1.9.0-rc2 (explain why)
> > >
> > > This vote will go for 72 hours.
> >
> >
>

Historical rebalance

[Seliverstov Igor]

Hi Igniters,

Currently I’m working on possible approaches how to implement historical 
rebalance (delta rebalance using WAL iterator) over MVCC caches.

The main difficulty is that MVCC writes changes on tx active phase while 
partition update version, aka update counter, is being applied on tx finish. 
This means we cannot start iteration over WAL right from the pointer where the 
update counter updated, but should include updates, which the transaction that 
updated the counter did.

These updates may be much earlier than the point where the update counter was 
updated, so we have to be able to identify the point where the first update 
happened.

The proposed approach includes:

1) preserve list of active txs, sorted by the time of their first update (using 
WAL ptr of first WAL record in tx)

2) persist this list on each checkpoint (together with TxLog for example)

4) send whole active tx list (transactions which were in active state at the 
time the node was crushed, empty list in case of graceful node stop) as a part 
of partition demand message.

4) find a checkpoint where the earliest tx exists in persisted txs and use 
saved WAL ptr as a start point or apply current approach in case the active tx 
list (sent on previous step) is empty

5) start iteration.

Your thoughts?

Regards,
Igor

Re: Historical rebalance

[Seliverstov Igor]

Ivan,

1) The list is saved on each checkpoint, wholly (all transactions in active
state at checkpoint begin).
We need whole the list to get oldest transaction because after
the previous oldest tx finishes, we need to get the following one.

2) I guess there is a description of how persistent storage works and how
it restores [1]

Vladimir,

the whole list of what we going to store on checkpoint (updated):
1) Partition counter low watermark (LWM)
2) WAL pointer of earliest active transaction write to partition at the
time the checkpoint have started
3) List of prepared txs with acquired partition counters (which were
acquired but not applied yet)

This way we don't need any additional info in demand message. Start point
can be easily determined using stored WAL "back-pointer".

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-LocalRecoveryProcess


вт, 27 нояб. 2018 г. в 11:19, Vladimir Ozerov :

> Igor,
>
> Could you please elaborate - what is the whole set of information we are
> going to save at checkpoint time? From what I understand this should be:
> 1) List of active transactions with WAL pointers of their first writes
> 2) List of prepared transactions with their update counters
> 3) Partition counter low watermark (LWM) - the smallest partition counter
> before which there are no prepared transactions.
>
> And the we send to supplier node a message: "Give me all updates starting
> from that LWM plus data for that transactions which were active when I
> failed".
>
> Am I right?
>
> On Fri, Nov 23, 2018 at 11:22 AM Seliverstov Igor 
> wrote:
>
> > Hi Igniters,
> >
> > Currently I’m working on possible approaches how to implement historical
> > rebalance (delta rebalance using WAL iterator) over MVCC caches.
> >
> > The main difficulty is that MVCC writes changes on tx active phase while
> > partition update version, aka update counter, is being applied on tx
> > finish. This means we cannot start iteration over WAL right from the
> > pointer where the update counter updated, but should include updates,
> which
> > the transaction that updated the counter did.
> >
> > These updates may be much earlier than the point where the update counter
> > was updated, so we have to be able to identify the point where the first
> > update happened.
> >
> > The proposed approach includes:
> >
> > 1) preserve list of active txs, sorted by the time of their first update
> > (using WAL ptr of first WAL record in tx)
> >
> > 2) persist this list on each checkpoint (together with TxLog for example)
> >
> > 4) send whole active tx list (transactions which were in active state at
> > the time the node was crushed, empty list in case of graceful node stop)
> as
> > a part of partition demand message.
> >
> > 4) find a checkpoint where the earliest tx exists in persisted txs and
> use
> > saved WAL ptr as a start point or apply current approach in case the
> active
> > tx list (sent on previous step) is empty
> >
> > 5) start iteration.
> >
> > Your thoughts?
> >
> > Regards,
> > Igor
>

Re: Historical rebalance

[Seliverstov Igor]

Vladimir,

I think I got your point,

It should work if we do the next:
introduce two structures: active list (txs) and candidate list (updCntr ->
txn pairs)

Track active txs, mapping them to actual update counter at update time.
On each next update put update counter, associated with previous update,
into a candidates list possibly overwrite existing value (checking txn)
On tx finish remove tx from active list only if appropriate update counter
(associated with finished tx) is applied.
On update counter update set the minimal update counter from the candidates
list as a back-counter, clear the candidate list and remove an associated
tx from the active list if present.
Use back-counter instead of actual update counter in demand message.

вт, 27 нояб. 2018 г. в 12:56, Seliverstov Igor :

> Ivan,
>
> 1) The list is saved on each checkpoint, wholly (all transactions in
> active state at checkpoint begin).
> We need whole the list to get oldest transaction because after
> the previous oldest tx finishes, we need to get the following one.
>
> 2) I guess there is a description of how persistent storage works and how
> it restores [1]
>
> Vladimir,
>
> the whole list of what we going to store on checkpoint (updated):
> 1) Partition counter low watermark (LWM)
> 2) WAL pointer of earliest active transaction write to partition at the
> time the checkpoint have started
> 3) List of prepared txs with acquired partition counters (which were
> acquired but not applied yet)
>
> This way we don't need any additional info in demand message. Start point
> can be easily determined using stored WAL "back-pointer".
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-LocalRecoveryProcess
>
>
> вт, 27 нояб. 2018 г. в 11:19, Vladimir Ozerov :
>
>> Igor,
>>
>> Could you please elaborate - what is the whole set of information we are
>> going to save at checkpoint time? From what I understand this should be:
>> 1) List of active transactions with WAL pointers of their first writes
>> 2) List of prepared transactions with their update counters
>> 3) Partition counter low watermark (LWM) - the smallest partition counter
>> before which there are no prepared transactions.
>>
>> And the we send to supplier node a message: "Give me all updates starting
>> from that LWM plus data for that transactions which were active when I
>> failed".
>>
>> Am I right?
>>
>> On Fri, Nov 23, 2018 at 11:22 AM Seliverstov Igor 
>> wrote:
>>
>> > Hi Igniters,
>> >
>> > Currently I’m working on possible approaches how to implement historical
>> > rebalance (delta rebalance using WAL iterator) over MVCC caches.
>> >
>> > The main difficulty is that MVCC writes changes on tx active phase while
>> > partition update version, aka update counter, is being applied on tx
>> > finish. This means we cannot start iteration over WAL right from the
>> > pointer where the update counter updated, but should include updates,
>> which
>> > the transaction that updated the counter did.
>> >
>> > These updates may be much earlier than the point where the update
>> counter
>> > was updated, so we have to be able to identify the point where the first
>> > update happened.
>> >
>> > The proposed approach includes:
>> >
>> > 1) preserve list of active txs, sorted by the time of their first update
>> > (using WAL ptr of first WAL record in tx)
>> >
>> > 2) persist this list on each checkpoint (together with TxLog for
>> example)
>> >
>> > 4) send whole active tx list (transactions which were in active state at
>> > the time the node was crushed, empty list in case of graceful node
>> stop) as
>> > a part of partition demand message.
>> >
>> > 4) find a checkpoint where the earliest tx exists in persisted txs and
>> use
>> > saved WAL ptr as a start point or apply current approach in case the
>> active
>> > tx list (sent on previous step) is empty
>> >
>> > 5) start iteration.
>> >
>> > Your thoughts?
>> >
>> > Regards,
>> > Igor
>>
>

Re: [VOTE] Creation dedicated list for github notifiacations

[Seliverstov Igor]

+1

вт, 27 нояб. 2018 г. в 14:45, Юрий :

> +1
>
> вт, 27 нояб. 2018 г. в 11:22, Andrey Mashenkov  >:
>
> > +1
> >
> > On Tue, Nov 27, 2018 at 10:12 AM Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Plus this dedicated list should be properly documented in wiki,
> > mentioning
> > > it in How to Contribute [1] or in Make Teamcity Green Again [2] would
> be
> > a
> > > good idea.
> > >
> > > [1]
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Make+Teamcity+Green+Again
> > >
> > > On Tue, Nov 27, 2018 at 9:51 AM Павлухин Иван 
> > wrote:
> > >
> > > > +1
> > > > вт, 27 нояб. 2018 г. в 09:22, Dmitrii Ryabov  >:
> > > > >
> > > > > 0
> > > > > вт, 27 нояб. 2018 г. в 02:33, Alexey Kuznetsov <
> > akuznet...@apache.org
> > > >:
> > > > > >
> > > > > > +1
> > > > > > Do not forget notification from GitBox too!
> > > > > >
> > > > > > On Tue, Nov 27, 2018 at 2:20 AM Zhenya
>  > >
> > > > wrote:
> > > > > >
> > > > > > > +1, already make it by filers.
> > > > > > >
> > > > > > > > This was discussed already [1].
> > > > > > > >
> > > > > > > > So, I want to complete this discussion with moving outside
> > > dev-list
> > > > > > > > GitHub-notification to dedicated list.
> > > > > > > >
> > > > > > > > Please start voting.
> > > > > > > >
> > > > > > > > +1 - to accept this change.
> > > > > > > > 0 - you don't care.
> > > > > > > > -1 - to decline this change.
> > > > > > > >
> > > > > > > > This vote will go for 72 hours.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Time-to-remove-automated-messages-from-the-devlist-td37484i20.html
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alexey Kuznetsov
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
>
> --
> Живи с улыбкой! :D
>

Re: Historical rebalance

[Seliverstov Igor]

Ivan,

different transactions may be applied in different order on backup nodes.
That's why we need an active tx set
and some sorting by their update times. The idea is to identify a point in
time which starting from we may lost some updates.
This point:
   1) is the last acknowledged by all backups (including possible further
demander) update on timeline;
   2) have a specific update counter (aka back-counter) which we going to
start iteration from.

After additional thinking on, I've identified a rule:

There is two fences:
  1) update counter (UC) - this means that all updates, with less UC than
applied one, was applied on a node, having this UC.
  2) update in scope of TX - all updates are applied one by one
sequentially, this means that the fact of update guaranties the previous
update (statement) was finished on all TX participants.

Сombining them, we can say the next:

All updates, that was acknowledged at the time the last update of tx, which
updated UC, applied, are guaranteed to be presented on a node having such UC

We can use this rule to find an iterator start pointer.

ср, 28 нояб. 2018 г. в 20:26, Павлухин Иван :

> Guys,
>
> Another one idea. We can introduce additional update counter which is
> incremented by MVCC transactions right after executing operation (like
> is done for classic transactions). And we can use that counter for
> searching needed WAL records. Can it did the trick?
>
> P.S. Mentally I am trying to separate facilities providing
> transactions and durability. And it seems to me that those facilities
> are in different dimensions.
> ср, 28 нояб. 2018 г. в 16:26, Павлухин Иван :
> >
> > Sorry, if it was stated that a SINGLE transaction updates are applied
> > in a same order on all replicas then I have no questions so far. I
> > thought about reordering updates coming from different transactions.
> > > I have not got why we can assume that reordering is not possible. What
> > have I missed?
> > ср, 28 нояб. 2018 г. в 13:26, Павлухин Иван :
> > >
> > > Hi,
> > >
> > > Regarding Vladimir's new idea.
> > > > We assume that transaction can be represented as a set of
> independent operations, which are applied in the same order on both primary
> and backup nodes.
> > > I have not got why we can assume that reordering is not possible. What
> > > have I missed?
> > > вт, 27 нояб. 2018 г. в 14:42, Seliverstov Igor :
> > > >
> > > > Vladimir,
> > > >
> > > > I think I got your point,
> > > >
> > > > It should work if we do the next:
> > > > introduce two structures: active list (txs) and candidate list
> (updCntr ->
> > > > txn pairs)
> > > >
> > > > Track active txs, mapping them to actual update counter at update
> time.
> > > > On each next update put update counter, associated with previous
> update,
> > > > into a candidates list possibly overwrite existing value (checking
> txn)
> > > > On tx finish remove tx from active list only if appropriate update
> counter
> > > > (associated with finished tx) is applied.
> > > > On update counter update set the minimal update counter from the
> candidates
> > > > list as a back-counter, clear the candidate list and remove an
> associated
> > > > tx from the active list if present.
> > > > Use back-counter instead of actual update counter in demand message.
> > > >
> > > > вт, 27 нояб. 2018 г. в 12:56, Seliverstov Igor  >:
> > > >
> > > > > Ivan,
> > > > >
> > > > > 1) The list is saved on each checkpoint, wholly (all transactions
> in
> > > > > active state at checkpoint begin).
> > > > > We need whole the list to get oldest transaction because after
> > > > > the previous oldest tx finishes, we need to get the following one.
> > > > >
> > > > > 2) I guess there is a description of how persistent storage works
> and how
> > > > > it restores [1]
> > > > >
> > > > > Vladimir,
> > > > >
> > > > > the whole list of what we going to store on checkpoint (updated):
> > > > > 1) Partition counter low watermark (LWM)
> > > > > 2) WAL pointer of earliest active transaction write to partition
> at the
> > > > > time the checkpoint have started
> > > > > 3) List of prepared txs with acquired partition counters (which
> were
> > > > > acquired but not applied yet)
> > > > >
> > > > > This way

Re: Apache Ignite 2.7. Last Mile

[Seliverstov Igor]

Ivan,

Could you provide a bit more details?

I don't see any NPE among all available logs.

I don't think the issue is caused by changes in scope of IGNITE-7953.
The test fails both before

 and after

the
commit was merged to master with almost the same stack trace.

Regards,
Igor

чт, 29 нояб. 2018 г. в 18:43, Yakov Zhdanov :

> Vladimir, can you please take a look at
> https://issues.apache.org/jira/browse/IGNITE-10376?
>
> --Yakov
>

Re: Historical rebalance

[Seliverstov Igor]

ated WAL
> > record. This is where we start.
> >
> > So in terms of WAL we have: supplier[uc_backpointer <- cp(uc_current <=
> > demanter_uc_backpointer)] <- demander[uc_backpointer <- cp(last)]
> >
> > Now the most important - why it works :-)
> > 1) Transaction opeartions are sequential, so at the time of crash nodes
> > are *at most one operation ahead *each other
> > 2) Demander goes to the past and finds update counter which was current
> at
> > the time of last TX completed operation
> > 3) Supplier goes to the closest checkpoint in the past where this update
> > counter either doesn't exist or just appeared
> > 4) Transaction cannot be committed on supplier at this checkpoint, as it
> > would violate UC happens-before rule
> > 5) Tranasction may have not started yet on supplier at this point. If
> more
> > recent WAL records will contain *ALL* updates of the transaction
> > 6) Transaction may exist on supplier at this checkpoint. Thanks to p.1 we
> > must skip at most one operation. Jump back through supplier's checkpoint
> > backpointer is guaranteed to do this.
> >
> > Igor, do we have the same understanding here?
> >
> > Vladimir.
> >
> > On Thu, Nov 29, 2018 at 2:47 PM Seliverstov Igor 
> > wrote:
> >
> >> Ivan,
> >>
> >> different transactions may be applied in different order on backup
> nodes.
> >> That's why we need an active tx set
> >> and some sorting by their update times. The idea is to identify a point
> in
> >> time which starting from we may lost some updates.
> >> This point:
> >>1) is the last acknowledged by all backups (including possible
> further
> >> demander) update on timeline;
> >>2) have a specific update counter (aka back-counter) which we going
> to
> >> start iteration from.
> >>
> >> After additional thinking on, I've identified a rule:
> >>
> >> There is two fences:
> >>   1) update counter (UC) - this means that all updates, with less UC
> than
> >> applied one, was applied on a node, having this UC.
> >>   2) update in scope of TX - all updates are applied one by one
> >> sequentially, this means that the fact of update guaranties the previous
> >> update (statement) was finished on all TX participants.
> >>
> >> Сombining them, we can say the next:
> >>
> >> All updates, that was acknowledged at the time the last update of tx,
> >> which
> >> updated UC, applied, are guaranteed to be presented on a node having
> such
> >> UC
> >>
> >> We can use this rule to find an iterator start pointer.
> >>
> >> ср, 28 нояб. 2018 г. в 20:26, Павлухин Иван :
> >>
> >> > Guys,
> >> >
> >> > Another one idea. We can introduce additional update counter which is
> >> > incremented by MVCC transactions right after executing operation (like
> >> > is done for classic transactions). And we can use that counter for
> >> > searching needed WAL records. Can it did the trick?
> >> >
> >> > P.S. Mentally I am trying to separate facilities providing
> >> > transactions and durability. And it seems to me that those facilities
> >> > are in different dimensions.
> >> > ср, 28 нояб. 2018 г. в 16:26, Павлухин Иван :
> >> > >
> >> > > Sorry, if it was stated that a SINGLE transaction updates are
> applied
> >> > > in a same order on all replicas then I have no questions so far. I
> >> > > thought about reordering updates coming from different transactions.
> >> > > > I have not got why we can assume that reordering is not possible.
> >> What
> >> > > have I missed?
> >> > > ср, 28 нояб. 2018 г. в 13:26, Павлухин Иван :
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > Regarding Vladimir's new idea.
> >> > > > > We assume that transaction can be represented as a set of
> >> > independent operations, which are applied in the same order on both
> >> primary
> >> > and backup nodes.
> >> > > > I have not got why we can assume that reordering is not possible.
> >> What
> >> > > > have I missed?
> >> > > > вт, 27 нояб. 2018 г. в 14:42, Seliverstov Igor <
> >> gvvinbl...@gmail.com>:
> >> > > > >
> >> > > > > 

Re: [VOTE] Apache Ignite 2.7.0 RC1

[Seliverstov Igor]

+1

пт, 30 нояб. 2018 г., 9:59 Nikolay Izhikov nizhi...@apache.org:

> Igniters,
>
> We've uploaded a 2.7.0 release candidate to
>
> https://dist.apache.org/repos/dist/dev/ignite/2.7.0-rc1/
>
> Git tag name is 2.7.0-rc1
>
> This release includes the following changes:
>
> Apache Ignite In-Memory Database and Caching Platform 2.7
> -
>
> Ignite:
> * Added experimental support for multi-version concurrency control with
> snapshot isolation
>   - available for both cache API and SQL
>   - use CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT to enable it
>   - not production ready, data consistency is not guaranteed in case of
> node failures
> * Implemented Transparent Data Encryption based on JKS certificates
> * Implemented Node.JS Thin Client
> * Implemented Python Thin Client
> * Implemented PHP Thin Client
> * Ignite start scripts now support Java 9 and higher
> * Added ability to set WAL history size in bytes
> * Added SslContextFactory.protocols and SslContextFactory.cipherSuites
> properties to control which SSL encryption algorithms can be used
> * Added JCache 1.1 compliance
> * Added IgniteCompute.withNoResultCache method with semantics similar to
> ComputeTaskNoResultCache annotation
> * Spring Data 2.0 is now supported in the separate module
> 'ignite-spring-data_2.0'
> * Added monitoring of critical system workers
> * Added ability to provide custom implementations of ExceptionListener for
> JmsStreamer
> * Ignite KafkaStreamer was upgraded to use new KafkaConsmer configuration
> * S3 IP Finder now supports subfolder usage instead of bucket root
> * Improved dynamic cache start speed
> * Improved checkpoint performance by decreasing mark duration.
> * Added ability to manage compression level for compressed WAL archives.
> * Added metrics for Entry Processor invocations.
> * Added JMX metrics: ClusterMetricsMXBean.getTotalBaselineNodes and
> ClusterMetricsMXBean.getActiveBaselineNodes
> * Node uptime metric now includes days count
> * Exposed info about thin client connections through JMX
> * Introduced new system property IGNITE_REUSE_MEMORY_ON_DEACTIVATE to
> enable reuse of allocated memory on node deactivation (disabled by default)
> * Optimistic transaction now will be properly rolled back if waiting too
> long for a new topology on remap
> * ScanQuery with setLocal flag now checks if the partition is actually
> present on local node
> * Improved cluster behaviour when a left node does not cause partition
> affinity assignment changes
> * Interrupting user thread during partition initialization will no longer
> cause node to stop
> * Fixed problem when partition lost event was not triggered if multiple
> nodes left cluster
> * Fixed massive node drop from the cluster on temporary network issues
> * Fixed service redeployment on cluster reactivation
> * Fixed client node stability under ZooKeeper discovery
> * Massive performance and stability improvements
>
> Ignite .Net:
> * Add .NET Core 2.1 support
> * Added thin client connection failover
>
> Ignite C++:
> * Implemented Thin Client with base cache operations
> * Implemented smart affinity routing for Thin Client to send requests
> directly to nodes containing data when possible
> * Added Clang compiler support
>
> SQL:
> * Added experimental support for fully ACID transactional SQL with the
> snapshot isolation:
>   - use CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT to enable it
>   - a transaction can be started through native API (IgniteTransactions),
> thin JDBC driver or ODBC driver
>   - not production ready, data consistency is not guaranteed in case of
> node failures
> * Added a set of system views located in "IGNITE" schema to view cluster
> information (NODES, NODE_ATTRIBUTES, NODE_METRICS, BASELINE_NODES)
> * Added ability to create predefined SQL schemas
> * Added GROUP_CONCAT function support
> * Added string length constraint
> * Custom Java objects are now inlined into primary and secondary indexes
> what may significantly improve performance when AFFINITY_KEY is used
> * Added timeout to fail query execution in case it cannot be mapped to
> topology
> * Restricted number of cores allocated for CREATE INDEX by default to 4 to
> avoid contention on index tree Fixed transaction hanging during runtime
> error on commit.
> * Fixed possible memory leak when result set size is multiple of the page
> size
> * Fixed situation when data may be returned from cache partitions in LOST
> state even when PartitionLossPolicy doesn't permit it
> * Fixed "Caches have distinct sets of data nodes" during SQL JOIN query
> execution between REPLICATED and PARTITIONED caches
> * Fixed wrong result for SQL queries when item size exceeds the page size
> * Fixed error during SQL query from client node with the local flag set to
> "true"
> * Fixed handling UUID as a column type
>
> JDBC:
> * Implemented DataSource interface for the thin driver
>
> ODBC:
> * Added streaming mode support
> * Fixed crash 

Re: Apache Ignite 2.7. Last Mile

[Seliverstov Igor]

Null pointer there due to cache stop. Look at GridCacheContext#cleanup
(GridCacheContext.java:2050)
which is called by GridCacheProcessor#stopCache
(GridCacheProcessor.java:1372)

That's why at the time GridCacheMapEntry#touch (GridCacheMapEntry.java:5063)
 invoked there is no eviction manager.

This is a result of "normal" flow because message processing doesn't enter
cache gate like user API does.

пт, 30 нояб. 2018 г. в 10:26, Nikolay Izhikov :

> Ivan. Please, provide a link for a ticket with NPE stack trace attached.
>
> I've looked at IGNITE-10376 and can't see any attachments.
>
> пт, 30 нояб. 2018 г., 10:14 Ivan Fedotov ivanan...@gmail.com:
>
> > Igor,
> > NPE is available in a full log, now I also attached it in the ticket.
> >
> > IGNITE-7953
> > <
> >
> https://github.com/apache/ignite/commit/51a202a4c48220fa919f47147bd4889033cd35a8
> > >
> > was commited on the 15 October. I could not take a look on the
> > testAtomicOnheapTwoBackupAsyncFullSync before this date, because the
> oldest
> > test in the history on TC dates 12 November.
> >
> > So, I tested it locally and could not reproduce mentioned error.
> >
> > чт, 29 нояб. 2018 г. в 20:07, Seliverstov Igor :
> >
> > > Ivan,
> > >
> > > Could you provide a bit more details?
> > >
> > > I don't see any NPE among all available logs.
> > >
> > > I don't think the issue is caused by changes in scope of IGNITE-7953.
> > > The test fails both before
> > > <
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=2318582&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_ContinuousQuery4#testNameId3300126853696550025
> > > >
> > >  and after
> > > <
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=2345403&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_ContinuousQuery4#testNameId3300126853696550025
> > > >
> > > the
> > > commit was merged to master with almost the same stack trace.
> > >
> > > Regards,
> > > Igor
> > >
> > > чт, 29 нояб. 2018 г. в 18:43, Yakov Zhdanov :
> > >
> > > > Vladimir, can you please take a look at
> > > > https://issues.apache.org/jira/browse/IGNITE-10376?
> > > >
> > > > --Yakov
> > > >
> > >
> >
> >
> > --
> > Ivan Fedotov.
> >
> > ivanan...@gmail.com
> >
>

Re: Page eviction for in-memory mode with enabled MVCC

[Seliverstov Igor]

Roman,

I would prefer first option.

The fact user uses MVCC says he needs some more strict guaranties which
cannot meet in other modes.
I would rollback both txs in case we cannot provide such guaranties.

Regards,
Igor

пт, 14 дек. 2018 г. в 15:36, Roman Kondakov :

> Vladimir,
>
> I was thinking about your proposal to not evict locked and recent (the
> transaction that created the record is still active) entries from the
> cache. Let's imagine next situation: we have almost full memory and two
> transactions:
>
> 1. txA: "SELECT * FOR UPDATE"
>
> 2. txB: "INSERT ...many keys here..."
>
> In this case txA locks all entries in the cache, and therefore we cannot
> evict any of them. If then txB is trying to add a lot of data, it lead
> us to the OOM situation, which user is trying to avoid using cache
> evictions.
>
> I see two ways how to deal with this issue:
>
> 1. Allow OOM in MVCC caches with configured evictions and warn user
> about it in the docs.
>
> 2. Give up with the repeatable read guaranties in case of evictions for
> MVCC caches and warn users about it in the documentation.
>
> Second variant looks better for me because user may not expect OOM when
> he has configured eviction policy for cache.
>
> What do you think?
>
>
> --
> Kind Regards
> Roman Kondakov
>
> On 13.12.2018 22:33, Vladimir Ozerov wrote:
> > It's hard to believe that entries are not locked on backups, because we
> > wrtite data right away. Even if it so, it should be very easy to fix -
> just
> > do not evict and entry if it was created or deleted by currently active
> > transaction.
> >
> > On Thu, Dec 13, 2018 at 10:28 PM Roman Kondakov
> 
> > wrote:
> >
> >> Vladimir,
> >>
> >> we do not lock entries on backups when MVCC is enabled and therefore we
> >> don't avoid entry eviction from backup by locking. So, your first
> >> scenario with primary stop is still relevant.
> >>
> >>
> >> --
> >> Kind Regards
> >> Roman Kondakov
> >>
> >> On 13.12.2018 22:14, Vladimir Ozerov wrote:
> >>> No, I mean that we should think about what kind of guarantees it
> >> possible.
> >>> My proposal was to prevent evictions of locked entries. This way we can
> >> say
> >>> users: "if you want true REPEATABLE_READ when evictions are enabled,
> then
> >>> make sure to lock entries on every access". This effectively means that
> >> all
> >>> SELECT's should be replaced with "SELECT FOR UPDATE".
> >>>
> >>> On Thu, Dec 13, 2018 at 10:09 PM Roman Kondakov
> >> 
> >>> wrote:
> >>>
>  Vladimir,
> 
>  correct me please if i misunderstood your thought. So, if eviction is
>  not about a consistency at all, we may evict keys in any way because
>  broken repeatable read semantics is not the biggest problem here. But
> we
>  should add some notes about it to user documentation. Right?
> 
> 
>  --
>  Kind Regards
>  Roman Kondakov
> 
>  On 13.12.2018 17:45, Vladimir Ozerov wrote:
> > Roman,
> >
> > I would start with the fact that eviction can never be consistent
> >> unless
>  it
> > utilizes atomic broadcast protocol, which is not the case for Ignite.
> >> In
> > Ignite entries on node are evicted independently.
> >
> > So you may easily get into situation like this:
> > 1) Start a cache with 1 backup and FULL_SYNC mode
> > 2) Put a key to primary node
> > 3) Stop primary
> > 4) Try reading from new primary and get null because key was evicted
> > concurrently
> >
> > Or:
> > 1) Start a transaction in PESSIMISTIC/READ_COMMITTED mode
> > 2) Read a key, get value
> > 3) Read the same key again, get null
> >
> > So in reality the choice is not between consistent and inconsistent
> > behavior, but rather about degree of inconsistency. Any solution is
> > possible as long as we can explain it to the user. E.g. "do not
> evict a
>  key
> > if it is either write-locked".
> >
> >
> > On Thu, Dec 13, 2018 at 5:19 PM Vladimir Ozerov <
> voze...@gridgain.com>
> > wrote:
> >
> >> Andrey,
> >>
> >> We will not be able to cache the whole data set locally, as it
>  potentially
> >> lead to OOME. We will have this only as an option and only for
> non-SQL
> >> updates. Thus, similar semantics is not possible.
> >>
> >> On Thu, Dec 13, 2018 at 4:56 PM Andrey Mashenkov <
> >> andrey.mashen...@gmail.com> wrote:
> >>
> >>> Roman,
> >>>
> >>> We have a ticket to improve repeatable_read mode [1] via caching
>  entries
> >>> locally.
> >>> This should make mvcc transaction repeatable_read semantic similar
> to
> >>> non-mvcc Txs
> >>> and allow us to implement eviction in correct way.
> >>>
> >>> Another way is to introduce mvcc shared (read) entry locks and
> evict
>  only
> >>> entries if no one hold any lock on it,
> >>> but this looks tricky and error prone as your first one as it may
> >> lead
>  to
> >>> evic

Re: Suggestion to improve deadlock detection

[Seliverstov Igor]

Ivan,

I would prefer forward-only implementation even knowing it allows false
positive check results.

Why I think so:

1) From my experience, when we have any future is waiting for reply, we
have to take a failover into consideration.
Usually failover implementations are way more complex than an initial
algorithm itself.
Forward-only version doesn't need any failover implementation as it expects
failed probes (the probes didn't face a deadlock ).

2) Simple lightweight feature implementation is a really good point to
start - it may be extended if needed but really often such extending
doesn't need at all.

3) Any implementation allow false positive result - we are working with
distributed system, there are a bunch of processes happens at the same time
and,
for example, concurrently happened rollback on timeout may solve a deadlock
but probe is finished at this time, so- there is a rollback on deadlock
also as a result.

4) The described case (when false positive result is probable) isn't usual
but, potentially, extremely rare, I don't think we should solve it since it
doesn't break the grid.

Regards,
Igor

вт, 18 дек. 2018 г. в 17:57, Павлухин Иван :

> Hi folks,
>
> During implementation of edge-chasing deadlock detection algorithm in
> scope of [1] it has been realized that basically we have 2 options for
> "chasing" strategy. I will use terms Near when GridNearTxLocal is
> assumed and Dht when GridDhtTxLocal (tx which updates primary
> partition). So, here is 2 mentioned strategies:
> 1. Send initial probe when Dht tx blocks waiting for another tx to
> release a key. Initial probe is sent from waiting Dht tx to Near tx
> holding a lock. If receiving Near tx is blocked as well then it relays
> the probe to Dht tx it awaits response from. So, the probe is
> traveling like Dht -> Near -> Dht -> Near -> ... Let's call the
> approach "forward-only".
> 2. Send probes only between Near transactions. This approach requires
> additional request-response call which Near tx issues to Dht node to
> check if tx sending a request is waiting for another tx. The call
> returns identifier of a Near tx blocking tx sending a request (if
> sending tx is in fact blocked). Then waiting Near tx relays a probe to
> blocked Near tx. Let's call that approach "lock-checking".
>
> I would like to define several points to consider while comparing
> approaches (please point out more if I miss something):
> 1. Correctness. Especially regarding reporting false deadlocks.
> 2. Extensibility. Rollback to savepoint and generalization for classic
> transactions should be considered.
> 3. Messaging overhead.
> 4. Simplicity of implementation.
>
> You can find an implementation of "lock-checking" approach in PR
> attached to the ticket [1]. Currently it works correctly only for
> transactions obeying strict two-phase locking (2PL). It is fairly easy
> to adopt PR to implement "forward-only" approach. Which will work
> correct in conditions of 2PL. "lock-checking" algorithm uses 1.5 times
> more messages than "forward-only". Implementation of "forward-only"
> algorithm is simpler in a sense that it does not require introducing
> lock-wait-check messages and future. But as for me it does not look as
> big deal. I think that implementation efforts for both approaches are
> almost equal so far.
>
> But corner stone here is an extensibility. Let's imagine that we are
> going to implement rollback to savepoint. One suggested approach to
> extend "forward-only" approach for that case is invalidating sent
> probe. Basically, a tx initiated deadlock check assigns unique id to a
> probe. And this probe id is invalidated when the tx wakes up from
> wait. If that tx receives back a probe with invalidated id it simply
> discards it. If id is not invalidated it means a deadlock. But false
> deadlock still can be detected. I will provide an example when it does
> not work correctly.
>
> Please note, that our transactions can request multiple locks
> simultaneously (so-called AND model). Also rollback to savepoint
> releases locks obtained after creating a savepoint. Let's use
> following notation. "*" means savepoint. "^" means rollback to
> previous savepoint. "!" means acquiring a lock. "?" means waiting for
> a lock. "&" means requesting multiple locks simultaneously. See the
> following transaction execution scenario for 3 transactions and 3
> keys:
> TA| TB |   TC
> |  |   *
> |  1!  |  2!
> 1? & 3! |   |
> |  2? |
> |  |  ^
> |  |  3!
>
> We can notice that suggested scenario does not introduce deadlock
> because locks are requested in consistent order among all
> transactions. But in different moments there exist waiting relations
> TA w TB, TB w TC, TC w TA. So, we suspect a false deadlock could be
> detected here. Also one can notice that a relation TC w TA is
> established after TB w TC is destroyed. Messages on a timeline diagram
>

Null cache names

[Seliverstov Igor]

Hi Roman,

As far as I inderstand you're the author of the Redis protocol
implementation.

Currently I'm working on a task to prohibit null names for caches and I've
found that current implementation implies using of default (aka null named)
cache as a Redis database.

So, I need your assistance to implement Redis databases and mappings them
to particular caches.

Is it in your plans to do it in near time?

If not I'll appriciate your thoughts on how to do it properly)

Regards,
Igor

Re: Null cache names

[Seliverstov Igor]

Hi Roman,

The ticket number is IGNITE-3488.

It's planned not to have null-named or default caches at all. All the
caches must be defined explicitly or via templates and have names. The
current implementation uses a cache with null name, so, we need some
configuration property to define which cache to use or mapping in case of
using Redis databases.

Regards,
Igor

22 апр. 2017 г. 8:47 пользователь "Roman Shtykh" 
написал:

> Hi Igor,
> The current implementation supports only STRING data type of Redis. In
> addition, AFAIK, scaling Redis per dataset is normally done via running
> separate instances of Redis for each dataset. Therefore my choice was
> storing to the default cache. It looks fine from Redis' perspective, but
> probably not from Ignite's.For other data types like HASH or SORTED SET, I
> planned to specify cache name by key and treat value inside as Ignite's
> key-values.# Redis has a notion of 'database' and you can switch between
> them, but they can be referred only by the number, and limited to 16
> databases... Not very useful, IMO.
> If you still have the default cache, the current Redis integration should
> work as is (I have to recheck it, what is the JIRA ticket for the null
> cache issue?)
> Do you just want to be sure your changes don't affect the Redis
> integration, or do you want to extend it to switch between caches?
> Roman
>
> On Friday, April 21, 2017 8:17 PM, Seliverstov Igor <
> gvvinbl...@gmail.com> wrote:
>
>
>  Hi Roman,
>
> As far as I inderstand you're the author of the Redis protocol
> implementation.
>
> Currently I'm working on a task to prohibit null names for caches and I've
> found that current implementation implies using of default (aka null named)
> cache as a Redis database.
>
> So, I need your assistance to implement Redis databases and mappings them
> to particular caches.
>
> Is it in your plans to do it in near time?
>
> If not I'll appriciate your thoughts on how to do it properly)
>
> Regards,
> Igor
>
>
>

Re: Null cache names

[Seliverstov Igor]

Dear Igniters,

Seems we have almost the same issue with Memcached protocol.

There is an ability to define a cache name via operation extras message
part (
https://github.com/memcached/memcached/wiki/BinaryProtocolRevamped#packet-structure)
but it looks a bit complicated from my point of view...

Different client implementations might provide such functionality or not (I
mean an additional info in an operation extras), so, potential user might
have some difficultes using Ignite as a Memcached server because of this
ignite-specific message part becomes mandatory.

An alternative (an the best way, I think) is introducing a configuration
property to define which cache to use in case it hasn't be defined in a
message.

I'll appreciate any thoughts on that.

Regards,
Igor

2017-04-24 12:43 GMT+03:00 Roman Shtykh :

> Vladimir,
> Probably we may set the cache name via https://redis.io/commands/
> client-setname, save it and use until the user specifies another name.
> #That will be the name for the default cache (mainly for STRING data). For
> other data types, like hashes (https://redis.io/topics/data-types), I am
> thinking about putting data into caches specified by key.
> Or use https://redis.io/commands/config-set CONFIG SET DFLT_CACHE
> cache_name,and save cache name somewhere in Ignite cluster (what is the
> proper place to store such info?).
> For that, we have to implement one of the above-mentioned commands.
> What do you think?
>
>
>
> On Monday, April 24, 2017 4:34 PM, Vladimir Ozerov <
> voze...@gridgain.com> wrote:
>
>
>  Roman,
> Is it possible to define a kind of property in Redis connection string (or
> property map) for this purpose? IMO ideally we should "externalize" cache
> name somehow, so that it can be changed with no changes to user's code.
>
> Alex,
> Not sure if this is a good idea as you will end up with PARTITIONED cache
> without backups with no ability to change that.
>
> On Mon, Apr 24, 2017 at 9:35 AM, Alexey Kuznetsov 
> wrote:
>
> > Roman,
> >
> > Just as idea, how about in case of user does not configured "REDIS_CACHE"
> >  then create it via ignite.getOrCreateCache(new
> > CacheConfiguration("REDIS_CACHE"))
> > and prin warning to log "REDIS_CACHE not configured, using default
> > partitioned cache".
> >
> > What do you think?
> >
> > On Mon, Apr 24, 2017 at 10:26 AM, Roman Shtykh  >
> > wrote:
> >
> > > Denis, Igor,
> > > What can be done now
> > > 1. create a default cache name for Redis data, e.g. "REDIS_CACHE" that
> > has
> > > to be configured explicitly in xml file (as it is done with other
> caches)
> > > by a user if he/she needs Redis protocol.
> > > 2. Force users to specify cache names as prefix to their keys, so that
> we
> > > can parse and switch between caches.
> > > The 1st one is a very quick fix I can do today. This can be extended in
> > > future to have a separate cache for each data type.
> > > Roman
> > >
> > >
> > >On Monday, April 24, 2017 12:15 AM, Denis Magda <
> dma...@gridgain.com
> > >
> > > wrote:
> > >
> > >
> > >  Roman, would you suggest a quick solution for the redis integration or
> > > even
> > > implement it in the nearest couple of days? We need to change the
> defaul
> > > cache name in 2.0. Otherwise, it can be done only in an year or so in
> > 3.0.
> > >
> > > Denis
> > >
> > > On Sunday, April 23, 2017, Seliverstov Igor 
> > wrote:
> > >
> > > > Hi Roman,
> > > >
> > > > The ticket number is IGNITE-3488.
> > > >
> > > > It's planned not to have null-named or default caches at all. All the
> > > > caches must be defined explicitly or via templates and have names.
> The
> > > > current implementation uses a cache with null name, so, we need some
> > > > configuration property to define which cache to use or mapping in
> case
> > of
> > > > using Redis databases.
> > > >
> > > > Regards,
> > > > Igor
> > > >
> > > > 22 апр. 2017 г. 8:47 пользователь "Roman Shtykh"
> > >  > > > >
> > > > написал:
> > > >
> > > > > Hi Igor,
> > > > > The current implementation supports only STRING data type of Redis.
> > In
> > > > > addition, AFAIK, scaling Redis per dataset is normally done via
> > running
> > > > > separate insta

Re: Null cache names

[Seliverstov Igor]

Roman,

Why can't we use just some kind of mapping to bind Redis database ID to
cache name and define it via xml or programmatically?

In this case 0 database will be mapped to a default cache, so that we won't
break abstraction in Redis terms.

Regards,
Igor

2017-04-24 12:43 GMT+03:00 Roman Shtykh :

> Vladimir,
> Probably we may set the cache name via https://redis.io/commands/
> client-setname, save it and use until the user specifies another name.
> #That will be the name for the default cache (mainly for STRING data). For
> other data types, like hashes (https://redis.io/topics/data-types), I am
> thinking about putting data into caches specified by key.
> Or use https://redis.io/commands/config-set CONFIG SET DFLT_CACHE
> cache_name,and save cache name somewhere in Ignite cluster (what is the
> proper place to store such info?).
> For that, we have to implement one of the above-mentioned commands.
> What do you think?
>
>
>
> On Monday, April 24, 2017 4:34 PM, Vladimir Ozerov <
> voze...@gridgain.com> wrote:
>
>
>  Roman,
> Is it possible to define a kind of property in Redis connection string (or
> property map) for this purpose? IMO ideally we should "externalize" cache
> name somehow, so that it can be changed with no changes to user's code.
>
> Alex,
> Not sure if this is a good idea as you will end up with PARTITIONED cache
> without backups with no ability to change that.
>
> On Mon, Apr 24, 2017 at 9:35 AM, Alexey Kuznetsov 
> wrote:
>
> > Roman,
> >
> > Just as idea, how about in case of user does not configured "REDIS_CACHE"
> >  then create it via ignite.getOrCreateCache(new
> > CacheConfiguration("REDIS_CACHE"))
> > and prin warning to log "REDIS_CACHE not configured, using default
> > partitioned cache".
> >
> > What do you think?
> >
> > On Mon, Apr 24, 2017 at 10:26 AM, Roman Shtykh  >
> > wrote:
> >
> > > Denis, Igor,
> > > What can be done now
> > > 1. create a default cache name for Redis data, e.g. "REDIS_CACHE" that
> > has
> > > to be configured explicitly in xml file (as it is done with other
> caches)
> > > by a user if he/she needs Redis protocol.
> > > 2. Force users to specify cache names as prefix to their keys, so that
> we
> > > can parse and switch between caches.
> > > The 1st one is a very quick fix I can do today. This can be extended in
> > > future to have a separate cache for each data type.
> > > Roman
> > >
> > >
> > >On Monday, April 24, 2017 12:15 AM, Denis Magda <
> dma...@gridgain.com
> > >
> > > wrote:
> > >
> > >
> > >  Roman, would you suggest a quick solution for the redis integration or
> > > even
> > > implement it in the nearest couple of days? We need to change the
> defaul
> > > cache name in 2.0. Otherwise, it can be done only in an year or so in
> > 3.0.
> > >
> > > Denis
> > >
> > > On Sunday, April 23, 2017, Seliverstov Igor 
> > wrote:
> > >
> > > > Hi Roman,
> > > >
> > > > The ticket number is IGNITE-3488.
> > > >
> > > > It's planned not to have null-named or default caches at all. All the
> > > > caches must be defined explicitly or via templates and have names.
> The
> > > > current implementation uses a cache with null name, so, we need some
> > > > configuration property to define which cache to use or mapping in
> case
> > of
> > > > using Redis databases.
> > > >
> > > > Regards,
> > > > Igor
> > > >
> > > > 22 апр. 2017 г. 8:47 пользователь "Roman Shtykh"
> > >  > > > >
> > > > написал:
> > > >
> > > > > Hi Igor,
> > > > > The current implementation supports only STRING data type of Redis.
> > In
> > > > > addition, AFAIK, scaling Redis per dataset is normally done via
> > running
> > > > > separate instances of Redis for each dataset. Therefore my choice
> was
> > > > > storing to the default cache. It looks fine from Redis'
> perspective,
> > > but
> > > > > probably not from Ignite's.For other data types like HASH or SORTED
> > > SET,
> > > > I
> > > > > planned to specify cache name by key and treat value inside as
> > Ignite's
> > > > > key-values.# Redis has a notion of 'database' and you c

ignite-pds and binary objects

[Seliverstov Igor]

Hi guys!

Seems the new persistance feature is not compatable with our default
configuration.

A simplest test where I store an object type, restart the grid and try to
get the object by its key fails with NPE:

java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:492)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl$2.metadata(CacheObjectBinaryProcessorImpl.java:174)
at
org.apache.ignite.internal.binary.BinaryContext.metadata(BinaryContext.java:1231)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.getOrCreateSchema(BinaryReaderExImpl.java:1987)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:283)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.(BinaryReaderExImpl.java:182)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.reader(BinaryObjectImpl.java:827)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:791)
at
org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:142)
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:273)
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:161)
at
org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:148)
at
org.apache.ignite.internal.processors.cache.GridCacheContext.unwrapBinaryIfNeeded(GridCacheContext.java:1713)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.setResult(GridPartitionedSingleGetFuture.java:648)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.localGet(GridPartitionedSingleGetFuture.java:441)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:327)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:212)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:204)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1460)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:130)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:528)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:526)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:821)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync(GridDhtAtomicCache.java:526)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4458)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4439)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1326)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:1180)

Guess it happens because we don't store bynary metadata (with
use-compact-footer=false everything works fine).

Any thoughts on that?

I suppose we have to store the binary metadata if the persistence is on.

Regards,
Igor

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

What do you, guys think about next warning?

[2017-08-25 17:17:04,718][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
System cache's MemoryPolicy size is configured to 40 MB. Use
MemoryConfiguration.systemCacheMemorySize property to change the setting.
[2017-08-25 17:17:04,718][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]

>>> Required RAM size is larger than total physical memory available for OS.
>>> Check your configuration to avoid swap partition usage.
>>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
settings.
>>> Physical memory [required=16397MB, available=15942MB]

[2017-08-25 17:17:04,726][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Peer class loading is enabled (disable it in production for performance and
deployment consistency reasons)
[2017-08-25 17:17:04,726][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
[2017-08-25 17:17:04,731][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
3-rd party licenses can be found at:
/home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses


2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :

> Agree, let's release new version including tickets mentioned by Denis and
> Ivan.
>
> --Yakov
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

An example of current impl:


[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Not enough memory for current process [required=29251MB, available=15942MB].
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Current settings:
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  HeapInit=250MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  HeapMax=3543MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  DefaultMemoryPolicySize=12753MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  SystemCacheInitialSize=40MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  SystemCacheMaxSize=100MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Other ignite instanses on the server require: 12853MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]


2017-08-25 17:40 GMT+03:00 Sergey Kozlov :

> I suppose we should not forget JVM heap size and suggest to reduce both
> options
>
> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan 
> wrote:
>
> > Igor, I would change the message. How about this:
> >
> > Required RAM size is larger than total physical memory available for OS.
> > > Please change MemoryConfiguration.WhichProperty and
> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
> for
> > > each node.
> >
> >
> > Also, can we calculate what the memory size allocated for each node
> should
> > be? In that case we should suggest it.
> >
> > D.
> >
> >
> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor 
> > wrote:
> >
> > > What do you, guys think about next warning?
> > >
> > > [2017-08-25 17:17:04,718][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> > > MemoryConfiguration.systemCacheMemorySize property to change the
> > setting.
> > > [2017-08-25 17:17:04,718][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > >
> > > >>> Required RAM size is larger than total physical memory available
> for
> > > OS.
> > > >>> Check your configuration to avoid swap partition usage.
> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> > > settings.
> > > >>> Physical memory [required=16397MB, available=15942MB]
> > >
> > > [2017-08-25 17:17:04,726][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Peer class loading is enabled (disable it in production for performance
> > and
> > > deployment consistency reasons)
> > > [2017-08-25 17:17:04,726][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> > > [2017-08-25 17:17:04,731][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > 3-rd party licenses can be found at:
> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> > >
> > >
> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
> > >
> > > > Agree, let's release new version including tickets mentioned by Denis
> > and
> > > > Ivan.
> > > >
> > > > --Yakov
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

This message appears on topology change in case the available memory is
exceeded

2017-08-25 19:47 GMT+03:00 Seliverstov Igor :

> An example of current impl:
>
>
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> current process [required=29251MB, available=15942MB].
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> MemoryConfiguration.systemCacheMaxSize and 
> MemoryConfiguration.defaultMemoryPolicySize
> to decrease memory allocated for each node.
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   DefaultMemoryPolicySize=
> 12753MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> SystemCacheInitialSize=40MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   SystemCacheMaxSize=100MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses on
> the server require: 12853MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
>
> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
>
>> I suppose we should not forget JVM heap size and suggest to reduce both
>> options
>>
>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan > >
>> wrote:
>>
>> > Igor, I would change the message. How about this:
>> >
>> > Required RAM size is larger than total physical memory available for OS.
>> > > Please change MemoryConfiguration.WhichProperty and
>> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
>> for
>> > > each node.
>> >
>> >
>> > Also, can we calculate what the memory size allocated for each node
>> should
>> > be? In that case we should suggest it.
>> >
>> > D.
>> >
>> >
>> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor > >
>> > wrote:
>> >
>> > > What do you, guys think about next warning?
>> > >
>> > > [2017-08-25 17:17:04,718][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > System cache's MemoryPolicy size is configured to 40 MB. Use
>> > > MemoryConfiguration.systemCacheMemorySize property to change the
>> > setting.
>> > > [2017-08-25 17:17:04,718][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > >
>> > > >>> Required RAM size is larger than total physical memory available
>> for
>> > > OS.
>> > > >>> Check your configuration to avoid swap partition usage.
>> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change
>> the
>> > > settings.
>> > > >>> Physical memory [required=16397MB, available=15942MB]
>> > >
>> > > [2017-08-25 17:17:04,726][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Peer class loading is enabled (disable it in production for
>> performance
>> > and
>> > > deployment consistency reasons)
>> > > [2017-08-25 17:17:04,726][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
>> > > [2017-08-25 17:17:04,731][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > 3-rd party licenses can be found at:
>> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>> > >
>> > >
>> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
>> > >
>> > > > Agree, let's release new version including tickets mentioned by
>> Denis
>> > and
>> > > > Ivan.
>> > > >
>> > > > --Yakov
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Sergey Kozlov
>> GridGain Systems
>> www.gridgain.com
>>
>
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

The message without logging layout:

Not enough memory for current process [required=29251MB, available=15942MB].
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.

Current settings:
   HeapInit=250MB
   HeapMax=3543MB
   DefaultMemoryPolicySize=12753MB
   SystemCacheInitialSize=40MB
   SystemCacheMaxSize=100MB

Other ignite instanses on the server require: 12853MB

I think it's make sense to describe what these numbers are consist of.

We simple say which parameters have an impact on how much memory the
instance needs and their (parameters) actual values.

Also we notice that more than one Ignite instance are ran on the server or
workstation and it also consumes memory.

25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
dsetrak...@apache.org> написал:

> Igor, what is this flood of WARN messaging coming after the text? Are we
> really going to print this whole thing out?
>
> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor 
> wrote:
>
> > This message appears on topology change in case the available memory is
> > exceeded
> >
> > 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
> >
> > > An example of current impl:
> > >
> > >
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> > > current process [required=29251MB, available=15942MB].
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> > defaultMemoryPolicySize
> > > to decrease memory allocated for each node.
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
>  DefaultMemoryPolicySize=
> > > 12753MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > SystemCacheInitialSize=40MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
>  SystemCacheMaxSize=100MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
> on
> > > the server require: 12853MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > >
> > >
> > > 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
> > >
> > >> I suppose we should not forget JVM heap size and suggest to reduce
> both
> > >> options
> > >>
> > >> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > >> >
> > >> wrote:
> > >>
> > >> > Igor, I would change the message. How about this:
> > >> >
> > >> > Required RAM size is larger than total physical memory available for
> > OS.
> > >> > > Please change MemoryConfiguration.WhichProperty and
> > >> > > MemoryPolicyConfiguration.WhichProperty to decrease memory
> > allocated
> > >> for
> > >> > > each node.
> > >> >
> > >> >
> > >> > Also, can we calculate what the memory size allocated for each node
> > >> should
> > >> > be? In that case we should suggest it.
> > >> >
> > >> > D.
> > >> >
> > >> >
> > >> > On Fri, Aug 25, 2017 at 7:20 AM, Seli

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

One more example of possible warning:

-
Excessive memory usage by Ignite node process (performance may drop)
[requested=44613MB, available=15942MB].

Please tune the folowing settings as suggested:
  MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
  MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
  MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
  MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB

Current settings:
  Java Heap  maxSize: 3543MB
  Java Heap initSize: 250MB
  MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
  MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
  MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
  MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
  The overall expected memory usage by all Ignite nodes on the host: 44613MB
-

Your thoughts?

2017-08-28 5:06 GMT+03:00 Denis Magda :

> Guys,
>
> ML lib profile is missing in 2.1 release! That must be fixed and rolled
> out in this emergency release:
> https://issues.apache.org/jira/browse/IGNITE-6193 <
> https://issues.apache.org/jira/browse/IGNITE-6193>
>
> Oleg, Yuri, please step in and handle the issue.
>
> BTW, who is considered to be the release manager of this release?
>
> —
> Denis
>
> > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan 
> wrote:
> >
> > I like the format proposed by Denis, very clear.
> >
> > However, I also do not understand why a user should change the size of
> some
> > system cache. How would a user ever know what value to put there? This
> > value should be configured by Ignite automatically.
> >
> > D.
> >
> > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda  wrote:
> >
> >> Igor,
> >>
> >> Let me suggest this format.
> >>
> >> -
> >> Excessive memory usage by Ignite node process (performance may drop)
> >> [requested=29251MB, available=15942MB]
> >>
> >> Please tune the following settings:
> >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> >> value]
> >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> >> value]
> >>
> >> Current settings:
> >>   [DefaultMemoryPolicySize = value]
> >>   [{policy_name_1} size = value]
> >>   [{policy_name_1} size = value]
> >>   SystemCacheInitialSize = value
> >>   SystemCacheMaxSize = value
> >>   Java Heap Init Size = value
> >>   Java Heap Max Size = value
> >>
> >> The overall memory usage by all Ignite nodes on the host: value
> >> ---
> >>
> >> Records in […] are optional. If custom memory policy is not set or the
> >> default memory policy is overridden the output will miss some of the
> rows.
> >>
> >> As for systemCacheMaxSize, it should be show ONLY if the parameter was
> set
> >> explicitly by user code. Otherwise, the platform should be wise enough
> to
> >> instantiate it properly depending on the host memory usage.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor 
> >> wrote:
> >>>
> >>> The message without logging layout:
> >>>
> >>> Not enough memory for current process [required=29251MB,
> >> available=15942MB].
> >>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> allocated
> >>> for each node.
> >>>
> >>> Current settings:
> >>>  HeapInit=250MB
> >>>  HeapMax=3543MB
> >>>  DefaultMemoryPolicySize=12753MB
> >>>  SystemCacheInitialSize=40MB
> >>>  SystemCacheMaxSize=100MB
> >>>
> >>> Other ignite instanses on the server require: 12853MB
> >>>
> >>> I think it's make sense to describe what these numbers are consist of.
> >>>
> >>> We simple say which parameters have an impact on how much memory the
> >>> instance needs and their (parameters) actual values.
> >>>
> >>> Also we notice that more than one Ignite instance are ran on the server
> >> or
> >>> workstation and it also consumes memory.
> >>>
> >>> 2

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

The suggestion here is based on initial settings, and it's so because there
is no other nodes on the host in the example.

The algorithm tries to preserve the original ratio of memory policies
keeping numbers reasonable (for example after some thresshold it will
suggest not to use several memory policies if there is not enough of memory
for all of them) and taking into consideration nodes count on the host,
each jvm heap, needed memory for OS, etc

2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :

> Looks good, but why in the example provided are we suggesting 8GB? 2 nodes
> with 8GB will completely exhaust the available memory. I would suggest 6 or
> 7GB.
>
> Also, why 100MB for default policy. Anything under 1GB seems too small.
>
> Can you please comment?
>
> D.
>
> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor 
> wrote:
>
> > One more example of possible warning:
> >
> > -
> > Excessive memory usage by Ignite node process (performance may drop)
> > [requested=44613MB, available=15942MB].
> >
> > Please tune the folowing settings as suggested:
> >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >
> > Current settings:
> >   Java Heap  maxSize: 3543MB
> >   Java Heap initSize: 250MB
> >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >   The overall expected memory usage by all Ignite nodes on the host:
> > 44613MB
> > -
> >
> > Your thoughts?
> >
> > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> >
> > > Guys,
> > >
> > > ML lib profile is missing in 2.1 release! That must be fixed and rolled
> > > out in this emergency release:
> > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > >
> > > Oleg, Yuri, please step in and handle the issue.
> > >
> > > BTW, who is considered to be the release manager of this release?
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > > wrote:
> > > >
> > > > I like the format proposed by Denis, very clear.
> > > >
> > > > However, I also do not understand why a user should change the size
> of
> > > some
> > > > system cache. How would a user ever know what value to put there?
> This
> > > > value should be configured by Ignite automatically.
> > > >
> > > > D.
> > > >
> > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda 
> > wrote:
> > > >
> > > >> Igor,
> > > >>
> > > >> Let me suggest this format.
> > > >>
> > > >> -
> > > >> Excessive memory usage by Ignite node process (performance may drop)
> > > >> [requested=29251MB, available=15942MB]
> > > >>
> > > >> Please tune the following settings:
> > > >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > > >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> > > >> value]
> > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> > > >> value]
> > > >>
> > > >> Current settings:
> > > >>   [DefaultMemoryPolicySize = value]
> > > >>   [{policy_name_1} size = value]
> > > >>   [{policy_name_1} size = value]
> > > >>   SystemCacheInitialSize = value
> > > >>   SystemCacheMaxSize = value
> > > >>   Java Heap Init Size = value
> > > >>   Java Heap Max Size = value
> > > >>
> > > >> The overall memory usage by all Ignite nodes on the host: value
> > > >> ---
> > > >>
> > > >> Records in […] are optional. If custom memory policy is not set or
> the
> >

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

Ok, the check happens at the node start time or on NODE_JOIN event

in general it looks like:

1) calculate expected used memory = heap max + system cache max + all
custom policies max + default policy size and put it into a node attribute

2) get total physycal memory, calculate expected safe to be used memory
amount (leave 4 gb min or 20% of available memory for OS)

3) if expected used memory + expected used memory of other nodes on the
host > than safe to be used memory amount, start calculating suggestions

4) Each ignite instance needs at least 512mb heap + 40mb system cache +
100mb default polycy, if available memory is less we cannot suggest
anything reasonable, print warning, stop calculation.

5) check heap size (shouldn't exceed 30% of available memory (total_memory
- reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
calculated value or 512MB minimal)

6) check if system cache size changed, suggest default value if it's so

7) in case 100 mb * policies count < available memory, suggest using
default policy with max size equals to remaining memory (available - heap -
system cache)

8) calculate new size for each memory policy ( it's user defined size *
(remaining / (all_policies_size * nodes_cnt)); in proportion to
remaining memory, devided by nodes number on the host or 100 mb minimal)

9) print suggestions



2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :

> Igor, can you please describe the algorithm with all the thresholds?
>
> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor 
> wrote:
>
> > The suggestion here is based on initial settings, and it's so because
> there
> > is no other nodes on the host in the example.
> >
> > The algorithm tries to preserve the original ratio of memory policies
> > keeping numbers reasonable (for example after some thresshold it will
> > suggest not to use several memory policies if there is not enough of
> memory
> > for all of them) and taking into consideration nodes count on the host,
> > each jvm heap, needed memory for OS, etc
> >
> > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > nodes
> > > with 8GB will completely exhaust the available memory. I would suggest
> 6
> > or
> > > 7GB.
> > >
> > > Also, why 100MB for default policy. Anything under 1GB seems too small.
> > >
> > > Can you please comment?
> > >
> > > D.
> > >
> > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> gvvinbl...@gmail.com>
> > > wrote:
> > >
> > > > One more example of possible warning:
> > > >
> > > > -
> > > > Excessive memory usage by Ignite node process (performance may drop)
> > > > [requested=44613MB, available=15942MB].
> > > >
> > > > Please tune the folowing settings as suggested:
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > >
> > > > Current settings:
> > > >   Java Heap  maxSize: 3543MB
> > > >   Java Heap initSize: 250MB
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > >   The overall expected memory usage by all Ignite nodes on the host:
> > > > 44613MB
> > > > -
> > > >
> > > > Your thoughts?
> > > >
> > > > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> > > >
> > > > > Guys,
> > > > >
> > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > rolled
> > > > > out in this emergency release:
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > >
> > > > > Oleg, Yuri, please step in and handle the issue.
> > > > >
> > > > > BTW, who is considered to be the release manager of this release?
> > > > >
> > > > > —
> > > > > Denis
> > > > 

MVCC and IgniteDataStreamer

[Seliverstov Igor]

Hi Igniters,

You know, we're currently developing fair MVCC for transactional caches (
https://issues-test.apache.org/jira/plugins/servlet/mobile#issue/IGNITE-4191
).

At now we're trying to make work IgniteDataStreamer with MVCC.

The problem is that MVCC requires linking between versions for checking
rows with visibility rules (see
https://cwiki.apache.org/confluence/display/IGNITE/Distributed+MVCC+And+Transactional+SQL+Design
for details). We cannot do that without write lock on a data row (there are
no guarantees the linking is consistent otherwise). But locks will affect
data streamer performance dramatically.

So, there are two ways, fast one and right one:

1) we introduce a special version which is lower than any other. All
streamed rows are written with this version. All other versions of row are
cleaned up. All pending transactions that involves these rows are marked as
rollback only. Repeatable read simanthyc is broken for reads (since initial
version is always visible, readers see data streamer dirty writes). User
has to ensure there is no other read/write operations while loading.

2) Data Streamer uses it's own mvcc version. All data streamer operations
become transactional. Data streamer acquires table lock before streaming
(write lock). Readers are not affected and see consistent snapshot while
data is loading.

Initially we're going to implement first approach (fast one) and as soon
table locks are introduced (there is an appropriate IEP) we'll do things
right.

What do you think?

Re: MVCC and IgniteDataStreamer

[Seliverstov Igor]

Yakov,

We can introduce several modes:

1) initial loading which replaces data (allowOverwrite=true) with initial 
version or leaves it as is (allowOverwrite=false) and requires exclusive table 
lock (fastest one)
2) continuous loading which has its own version and links the data as regular 
transaction (allowOverwrite=true) or leaves it as is (allowOverwrite=false), 
doesn’t affect concurrent readers but still requires write lock on a table 
(less fast than previous)
3) batch loading which acts as a sequence of regular transaction with all 
possible optimizations, doesn’t affect concurrent readers and writers, but 
causes possible lock conflicts with subsequent retries, links the data as 
regular transaction (allowOverwrite=true) or leaves it as is 
(allowOverwrite=false), doesn’t cause write conflicts (like READ_COMMITTED txs) 
(slowest one).

All the modes require table locks.

Your thoughts?

> 9 июля 2018 г., в 12:55, Yakov Zhdanov  написал(а):
> 
> Igor,
> 
> I can't say if I agree with any of the suggestions. I would like us to
> start from answering the question - what is data streamer used for?
> 
> First of all, for initial data loading. This should be super fast mode
> probably ignoring all transactional semantics, but providing certain
> guarantees for data passed into streamer to be loaded.
> 
> Second, for continuously streaming updates to some tables (from more than 1
> streamer) and running some analytics over data, probably, with some
> modifications from non-streamer side (user transactions). This way
> streamers should not rollback user txs or do any kind of unexpected
> visibility tricks. I think we can think of proper streamer tx on batch or
> key level.
> 
> Third case I see is a combination of the above - we stream portions of data
> to an existing table let's say once a day (which may be some market data
> after closing or offloaded operations data set) with or without any other
> concurrent non-streamer operations. This mode may involve table locks or do
> the same as 2nd mode which should be up to user to decide.
> 
> So, planned changes to streamer should support at least these 3 scenarios.
> What do you think?
> 
> Igniters, feel free sharing your thoughts on this. Question is pretty
> important for us.
> 
> --Yakov

Re: MVCC and IgniteDataStreamer

[Seliverstov Igor]

Ivan,

Anyway DataStreamer is the fastest way to deliver data to a data node, the 
question is how to apply it correctly.

I don’t thing we need one more tool, which 90% is the same as DataStreamer.

All we need is just to implement a couple of new stream receivers.

Regards,
Igor

> 13 июля 2018 г., в 9:56, Павлухин Иван  написал(а):
> 
> Hi Igniters,
> 
> I had a look into IgniteDataStreamer. As far as I understand, currently it
> just works incorrectly for MVCC tables. It appears as a blocker for
> releasing MVCC. The simplest thing is to refuse creating streamer for MVCC
> tables.
> 
> Next step could be hair splitting of related use cases. For me, initial
> load and continuous streaming look quite different cases and it is better
> to keep them separate at least at API level. Perhaps, it is better to
> separate API basing on user experience. For example, DataStreamer could be
> considered tool without surprises (which means leaving data always
> consistent, transactions). And let's say BulkLoader is a beast for fastest
> data loading but full of surprises. Such surprises could be locking tables,
> rolling back user transactions and so on. So, it is of very limited use
> (like initial load). Keeping API entities separate looks better for me than
> introducing multiple modes, because separated entities are easier for
> understanding and so less prone to user mistakes.
> 
> -- 
> Best regards,
> Ivan Pavlukhin