Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

On Wed, May 30, 2018 at 2:09 PM, Peter Ivanov  wrote:

> Dmitriy,
>
> What kind of Ubuntu installation do you have (and what version)?
> Is it WSL under Windows 10 or containerized (docker)?
>
>
Windows Ubuntu App -
https://docs.microsoft.com/en-us/windows/wsl/install-win10

D.

Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

Petr,

I have finally was able to install Ignite Debian package. But then I got
stuck at the very next instruction:


> *d@DmitriyPC:~$ sudo systemctl start
> apache-ignite@default-config.xmlFailed to connect to bus: No such file or
> directory*


D.

On Wed, May 30, 2018 at 7:40 AM, Petr Ivanov  wrote:

> Sometimes Ubuntu with requested and dependency packages recommends
> additional packages (which is controlled by corresponding directive in
> package).
> This option --no-install-recommends makes sure that these packages will be
> skipped and overall installation will be more compact and precise.
>
> There is no such option for YUM (as far as I am aware of).
>
>
>
> > On 30 May 2018, at 17:16, Dmitriy Setrakyan 
> wrote:
> >
> > Petr,
> >
> > Can you tell me what this flag does: "--no-install-recommends"?
> >
> > I am seeing that it is set for Debian, but not for RPM.
> >
> > D.
> >
> > On Wed, May 30, 2018 at 7:07 AM, Petr Ivanov 
> wrote:
> >
> >> Of course: https://apacheignite.readme.io/docs/getting-started#
> >> alternative-installation-options <https://apacheignite.readme.
> >> io/docs/getting-started#alternative-installation-options>
> >> Snippet "DEB repository setup"
> >>
> >>
> >>
> >>
> >>> On 30 May 2018, at 16:59, Dmitriy Setrakyan 
> >> wrote:
> >>>
> >>> Petr, can you provide a link to the Debian snippet on readme.io? I
> >> cannot
> >>> find it.
> >>>
> >>> On Wed, May 30, 2018 at 6:16 AM, Petr Ivanov 
> >> wrote:
> >>>
> >>>> Docker images for Apache Ignite [1] and Web Console [2] 2.5.0 are
> >> uploaded.
> >>>>
> >>>>
> >>>> [1] https://hub.docker.com/r/apacheignite/ignite/tags
> >>>> [2] https://hub.docker.com/r/apacheignite/web-console-
> standalone/tags/
> >>>>
> >>>>
> >>>>
> >>>>> On 30 May 2018, at 06:55, Petr Ivanov  wrote:
> >>>>>
> >>>>> Instructions for RPM Package Installation (snippet with repository
> >>>> configuration) at ignite.apache.org <http://ignite.apache.org/> are
> >>>> obsolete and should be replaced by apacheignite.readme.io <
> >>>> http://apacheignite.readme.io/> snippet.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 30 May 2018, at 02:57, Prachi Garg  >>>> pg...@gridgain.com>> wrote:
> >>>>>>
> >>>>>> Peter,
> >>>>>>
> >>>>>> I updated the RPM and DEB description on the downloads page[1]
> >>>> (following the instructions on the documentation [2]). Please check
> that
> >>>> the repositories and installation instructions are up-to-date.
> >>>>>>
> >>>>>> [1] https://ignite.apache.org/download.cgi#rpm-package <
> >>>> https://ignite.apache.org/download.cgi#rpm-package>
> >>>>>> [2] https://apacheignite.readme.io/docs/getting-started#
> >>>> section-rpm-deb-packages-installation <https://apacheignite.readme.
> >>>> io/docs/getting-started#section-rpm-deb-packages-installation>
> >>>>>>
> >>>>>> On Tue, May 29, 2018 at 4:47 PM, Dmitriy Setrakyan <
> >>>> dsetrak...@apache.org <mailto:dsetrak...@apache.org>> wrote:
> >>>>>> I just tried the Debian install. This is what I got after executing
> >>>> "sudo
> >>>>>> apt get" command:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> *E: The repository 'https://apache.org/dist/ignite/deb <
> >>>> https://apache.org/dist/ignite/deb>
> >>>>>>> <https://apache.org/dist/ignite/deb <https://apache.org/dist/
> >>>> ignite/deb>> apache-ignite Release' does not have a
> >>>>>>> Release file.N: Updating from such a repository can't be done
> >>>> securely, and
> >>>>>>> is therefore disabled by default.N: See apt-secure(8) manpage for
> >>>>>>> repository creation and user configuration details.*
> >>>>>>
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Tue, May 29, 2018 at 4:4

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

Dmitriy,

I think the behavior for offline and online iterator is fundamentally
different. I do not think it is OK to skip records during normal operation.
In my view, we should report an error and stop. However, when offline, it
is OK to report an error and continue in my view.

D.

On Wed, May 30, 2018 at 5:39 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Yakov,
>
> This problem is not for offline only, it applicable for all types of the
> iterators.
> In general, iterator does not know which class will be needed for
> deserialization.
>
> I guess we can expand WAL Iterator factory and provide a method for
> creating the iterator with some filter,
> which will accept predicate for skipping no interesting records. It will
> fix 1 problem.
>
>
> On Wed, May 30, 2018 at 3:18 PM, Yakov Zhdanov 
> wrote:
>
> > This is for offline WAL analysis. So skipping record with proper message
> is
> > also a solution. If it is possible, iterator should output a suggestion
> on
> > what is missing in classpath. Option to suppress warnings should also
> > present.
> >
> > Makes sense?
> >
> > And final question - did we look at similar utilities from other vendors?
> >
> > --Yakov
> >
>

Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

Petr,

Can you tell me what this flag does: "--no-install-recommends"?

I am seeing that it is set for Debian, but not for RPM.

D.

On Wed, May 30, 2018 at 7:07 AM, Petr Ivanov  wrote:

> Of course: https://apacheignite.readme.io/docs/getting-started#
> alternative-installation-options <https://apacheignite.readme.
> io/docs/getting-started#alternative-installation-options>
> Snippet "DEB repository setup"
>
>
>
>
> > On 30 May 2018, at 16:59, Dmitriy Setrakyan 
> wrote:
> >
> > Petr, can you provide a link to the Debian snippet on readme.io? I
> cannot
> > find it.
> >
> > On Wed, May 30, 2018 at 6:16 AM, Petr Ivanov 
> wrote:
> >
> >> Docker images for Apache Ignite [1] and Web Console [2] 2.5.0 are
> uploaded.
> >>
> >>
> >> [1] https://hub.docker.com/r/apacheignite/ignite/tags
> >> [2] https://hub.docker.com/r/apacheignite/web-console-standalone/tags/
> >>
> >>
> >>
> >>> On 30 May 2018, at 06:55, Petr Ivanov  wrote:
> >>>
> >>> Instructions for RPM Package Installation (snippet with repository
> >> configuration) at ignite.apache.org <http://ignite.apache.org/> are
> >> obsolete and should be replaced by apacheignite.readme.io <
> >> http://apacheignite.readme.io/> snippet.
> >>>
> >>>
> >>>
> >>>> On 30 May 2018, at 02:57, Prachi Garg  >> pg...@gridgain.com>> wrote:
> >>>>
> >>>> Peter,
> >>>>
> >>>> I updated the RPM and DEB description on the downloads page[1]
> >> (following the instructions on the documentation [2]). Please check that
> >> the repositories and installation instructions are up-to-date.
> >>>>
> >>>> [1] https://ignite.apache.org/download.cgi#rpm-package <
> >> https://ignite.apache.org/download.cgi#rpm-package>
> >>>> [2] https://apacheignite.readme.io/docs/getting-started#
> >> section-rpm-deb-packages-installation <https://apacheignite.readme.
> >> io/docs/getting-started#section-rpm-deb-packages-installation>
> >>>>
> >>>> On Tue, May 29, 2018 at 4:47 PM, Dmitriy Setrakyan <
> >> dsetrak...@apache.org <mailto:dsetrak...@apache.org>> wrote:
> >>>> I just tried the Debian install. This is what I got after executing
> >> "sudo
> >>>> apt get" command:
> >>>>
> >>>>
> >>>>>
> >>>>> *E: The repository 'https://apache.org/dist/ignite/deb <
> >> https://apache.org/dist/ignite/deb>
> >>>>> <https://apache.org/dist/ignite/deb <https://apache.org/dist/
> >> ignite/deb>> apache-ignite Release' does not have a
> >>>>> Release file.N: Updating from such a repository can't be done
> >> securely, and
> >>>>> is therefore disabled by default.N: See apt-secure(8) manpage for
> >>>>> repository creation and user configuration details.*
> >>>>
> >>>>
> >>>> D.
> >>>>
> >>>> On Tue, May 29, 2018 at 4:42 PM, Denis Magda  >> <mailto:dma...@apache.org>> wrote:
> >>>>
> >>>>> Andrey,
> >>>>>
> >>>>> Please help to generate 2.5 release HTML file and add it to Ignite
> >> website
> >>>>> SVN. Sergey K and Ilya S. should be able to assist with this.
> >>>>>
> >>>>> Sergey Chugunov and Ivan R., finalize the docs as soon as you can. We
> >>>>> can't announce the release and close it without having the docs done.
> >>>>>
> >>>>> --
> >>>>> Denis
> >>>>>
> >>>>>
> >>>>> On Tue, May 29, 2018 at 4:20 PM, Prachi Garg  >> <mailto:pg...@gridgain.com>> wrote:
> >>>>>
> >>>>>> Folks,
> >>>>>>
> >>>>>> I have released 2.5 documentation on readme.io <http://readme.io/>.
> >> Also, the downloads page
> >>>>>> on
> >>>>>> the website is updated with 2.5 release. Mauricio will work on
> >> updating
> >>>>>> the
> >>>>>> latest doc reference.
> >>>>>>
> >>>>>>
> >>>>>> Andrey,
> >>>>>>
> >>>>>> The following 

Re: Isn't SQL Streaming mode generic in 2.5?

[Dmitriy Setrakyan]

On Wed, May 30, 2018 at 5:40 AM, Igor Sapego  wrote:

> By the way, ODBC does not currently support streaming.
>

Igor, and when do you plan to add this support?

Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

Petr, can you provide a link to the Debian snippet on readme.io? I cannot
find it.

On Wed, May 30, 2018 at 6:16 AM, Petr Ivanov  wrote:

> Docker images for Apache Ignite [1] and Web Console [2] 2.5.0 are uploaded.
>
>
> [1] https://hub.docker.com/r/apacheignite/ignite/tags
> [2] https://hub.docker.com/r/apacheignite/web-console-standalone/tags/
>
>
>
> > On 30 May 2018, at 06:55, Petr Ivanov  wrote:
> >
> > Instructions for RPM Package Installation (snippet with repository
> configuration) at ignite.apache.org <http://ignite.apache.org/> are
> obsolete and should be replaced by apacheignite.readme.io <
> http://apacheignite.readme.io/> snippet.
> >
> >
> >
> >> On 30 May 2018, at 02:57, Prachi Garg  pg...@gridgain.com>> wrote:
> >>
> >> Peter,
> >>
> >> I updated the RPM and DEB description on the downloads page[1]
> (following the instructions on the documentation [2]). Please check that
> the repositories and installation instructions are up-to-date.
> >>
> >> [1] https://ignite.apache.org/download.cgi#rpm-package <
> https://ignite.apache.org/download.cgi#rpm-package>
> >> [2] https://apacheignite.readme.io/docs/getting-started#
> section-rpm-deb-packages-installation <https://apacheignite.readme.
> io/docs/getting-started#section-rpm-deb-packages-installation>
> >>
> >> On Tue, May 29, 2018 at 4:47 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org <mailto:dsetrak...@apache.org>> wrote:
> >> I just tried the Debian install. This is what I got after executing
> "sudo
> >> apt get" command:
> >>
> >>
> >> >
> >> > *E: The repository 'https://apache.org/dist/ignite/deb <
> https://apache.org/dist/ignite/deb>
> >> > <https://apache.org/dist/ignite/deb <https://apache.org/dist/
> ignite/deb>> apache-ignite Release' does not have a
> >> > Release file.N: Updating from such a repository can't be done
> securely, and
> >> > is therefore disabled by default.N: See apt-secure(8) manpage for
> >> > repository creation and user configuration details.*
> >>
> >>
> >> D.
> >>
> >> On Tue, May 29, 2018 at 4:42 PM, Denis Magda  <mailto:dma...@apache.org>> wrote:
> >>
> >> > Andrey,
> >> >
> >> > Please help to generate 2.5 release HTML file and add it to Ignite
> website
> >> > SVN. Sergey K and Ilya S. should be able to assist with this.
> >> >
> >> > Sergey Chugunov and Ivan R., finalize the docs as soon as you can. We
> >> > can't announce the release and close it without having the docs done.
> >> >
> >> > --
> >> > Denis
> >> >
> >> >
> >> > On Tue, May 29, 2018 at 4:20 PM, Prachi Garg  <mailto:pg...@gridgain.com>> wrote:
> >> >
> >> >> Folks,
> >> >>
> >> >> I have released 2.5 documentation on readme.io <http://readme.io/>.
> Also, the downloads page
> >> >> on
> >> >> the website is updated with 2.5 release. Mauricio will work on
> updating
> >> >> the
> >> >> latest doc reference.
> >> >>
> >> >>
> >> >> Andrey,
> >> >>
> >> >> The following still needs attention:
> >> >>
> >> >>1. Release notes for 2.5 on ignite.apache.org/releases/2.5.0/ <
> http://ignite.apache.org/releases/2.5.0/>
> >> >>2. Zookeper documentation -
> >> >>https://issues.apache.org/jira/browse/IGNITE-7900 <
> https://issues.apache.org/jira/browse/IGNITE-7900>
> >> >>3. Consistency check utilities docs -
> >> >>https://issues.apache.org/jira/browse/IGNITE-8524 <
> https://issues.apache.org/jira/browse/IGNITE-8524>
> >> >>
> >> >>
> >> >> Thanks,
> >> >> -Prachi
> >> >>
> >> >> On Tue, May 29, 2018 at 2:08 PM, Andrey Gura  <mailto:ag...@apache.org>> wrote:
> >> >>
> >> >> > Pavel,
> >> >> >
> >> >> > Thanks a lot!
> >> >> >
> >> >> >
> >> >> >
> >> >> > вт, 29 мая 2018 г., 22:14 Pavel Tupitsyn  <mailto:ptupit...@apache.org>>:
> >> >> >
> >> >> > > NuGet (.NET) packages pushed:
> >> >> > > https://www.nuget.org/packages?q=Apache.Ignite <
> https://www.nuget.org/packages?q=Apache.Ignite>
> >> >> > >
> >> >> > > On Tue, May 29, 2018 at 8:59 PM, Andrey Gura  <mailto:ag...@apache.org>>
> >> >> wrote:
> >> >> > >
> >> >> > >> Dmitry,
> >> >> > >>
> >> >> > >> First we should check docker images. Also we are waiting for
> >> >> > >> publishing of artifacts to Maven Central.
> >> >> > >>
> >> >> > >> On Tue, May 29, 2018 at 8:24 PM, Dmitriy Setrakyan
> >> >> > >> mailto:dsetrak...@apache.org>> wrote:
> >> >> > >> >
> >> >> > >> >
> >> >> > >> > On Tue, May 29, 2018 at 9:08 AM, Andrey Gura <
> ag...@apache.org <mailto:ag...@apache.org>>
> >> >> > wrote:
> >> >> > >> >>
> >> >> > >> >> RPM and DEB packages are published to bintray [1] [2]
> >> >> > >> >>
> >> >> > >> >> [1] https://bintray.com/apache/ignite-rpm <
> https://bintray.com/apache/ignite-rpm>
> >> >> > >> >> [2] https://bintray.com/apache/ignite-deb <
> https://bintray.com/apache/ignite-deb>
> >> >> > >> >
> >> >> > >> >
> >> >> > >> > Thanks, Andrey! Can we update the download page on the
> website?
> >> >> > >> >
> >> >> > >> > D.
> >> >> > >> >
> >> >> > >>
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
>
>

Re: Isn't SQL Streaming mode generic in 2.5?

[Dmitriy Setrakyan]

Alex,

STREAMING command is definitely a DML command. And by the way, the DML page
is not a mess. It documents all DML commands we have. I think you were
looking at something else.

What is the major difficulty of implementing STREAMING command on Ignite
SQL API? It should be even simpler than JDBC as all it should do is
delegate to the actual IgniteDataStreamer and that is it.

I keep repeating that we should stop adding handicapped features that work
only in some cases, but yet this trend continues.

Vladimir, can you comment?

D.

On Wed, May 30, 2018 at 5:30 AM, Alexander Paschenko <
apasche...@gridgain.com> wrote:

> Hi Denis,
>
> Currently there’s no way to do SQL streaming besides ODBC/JDBC - that is,
> there’s no other public API for it. I believe this is the first case when
> we’re looking at a command that is supported only via drivers and not Java
> API, so it’s an interesting question. Also streaming is no DDL, it doesn’t
> “define” any data. It could rather be deemed as DML command I think.
> That said, I think streaming docs being located at drivers’ section are
> fine, but we probably should add links to it from general data load
> section. Or maybe move it to an upper level - say, we have separate section
> for distributed joins, see no reason why streaming is different in this
> respect (it’s also an interesting and important feature that is about both
> SQL and Ignite specific details).
> Also DDL/DML doc pages are a mess - this one
> https://apacheignite-sql.readme.io/docs/ddl says nothing about new user
> management commands while this one
> https://apacheignite-sql.readme.io/docs/dml contains full set of DDL
> commands and not a single DML command (sic!) and doesn’t seem to be overly
> confident about its content judging by contradictions between its title and
> text (words DML/DDL are mixed). Both links are on main page of SQL docs and
> in dropdown menu. DML commands (SELECT, etc) seem to be available only via
> dropdown menu at the moment.
>
> Regards,
> Alex
>
> ср, 30 мая 2018 г. в 1:28, Denis Magda :
>
> > Vladimir, Alexander P., Igor,
> >
> > We've documented the streaming mode usage (SET STREAMING ON/OFF) on the
> > JDBC docs:
> > https://apacheignite-sql.readme.io/v2.5/docs/jdbc-
> driver#section-streaming
> >
> > But my guts feel it it's generic documentation that applies to all our
> SQL
> > interfaces and should go as a subpage of DDL section:
> > https://apacheignite-sql.readme.io/v2.5/docs/ddl
> >
> > Am I correct?
> >
> > --
> > Denis
> >
>

Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

I just tried the Debian install. This is what I got after executing "sudo
apt get" command:


>
> *E: The repository 'https://apache.org/dist/ignite/deb
> <https://apache.org/dist/ignite/deb> apache-ignite Release' does not have a
> Release file.N: Updating from such a repository can't be done securely, and
> is therefore disabled by default.N: See apt-secure(8) manpage for
> repository creation and user configuration details.*


D.

On Tue, May 29, 2018 at 4:42 PM, Denis Magda  wrote:

> Andrey,
>
> Please help to generate 2.5 release HTML file and add it to Ignite website
> SVN. Sergey K and Ilya S. should be able to assist with this.
>
> Sergey Chugunov and Ivan R., finalize the docs as soon as you can. We
> can't announce the release and close it without having the docs done.
>
> --
> Denis
>
>
> On Tue, May 29, 2018 at 4:20 PM, Prachi Garg  wrote:
>
>> Folks,
>>
>> I have released 2.5 documentation on readme.io. Also, the downloads page
>> on
>> the website is updated with 2.5 release. Mauricio will work on updating
>> the
>> latest doc reference.
>>
>>
>> Andrey,
>>
>> The following still needs attention:
>>
>>1. Release notes for 2.5 on ignite.apache.org/releases/2.5.0/
>>2. Zookeper documentation -
>>https://issues.apache.org/jira/browse/IGNITE-7900
>>3. Consistency check utilities docs -
>>https://issues.apache.org/jira/browse/IGNITE-8524
>>
>>
>> Thanks,
>> -Prachi
>>
>> On Tue, May 29, 2018 at 2:08 PM, Andrey Gura  wrote:
>>
>> > Pavel,
>> >
>> > Thanks a lot!
>> >
>> >
>> >
>> > вт, 29 мая 2018 г., 22:14 Pavel Tupitsyn :
>> >
>> > > NuGet (.NET) packages pushed:
>> > > https://www.nuget.org/packages?q=Apache.Ignite
>> > >
>> > > On Tue, May 29, 2018 at 8:59 PM, Andrey Gura 
>> wrote:
>> > >
>> > >> Dmitry,
>> > >>
>> > >> First we should check docker images. Also we are waiting for
>> > >> publishing of artifacts to Maven Central.
>> > >>
>> > >> On Tue, May 29, 2018 at 8:24 PM, Dmitriy Setrakyan
>> > >>  wrote:
>> > >> >
>> > >> >
>> > >> > On Tue, May 29, 2018 at 9:08 AM, Andrey Gura 
>> > wrote:
>> > >> >>
>> > >> >> RPM and DEB packages are published to bintray [1] [2]
>> > >> >>
>> > >> >> [1] https://bintray.com/apache/ignite-rpm
>> > >> >> [2] https://bintray.com/apache/ignite-deb
>> > >> >
>> > >> >
>> > >> > Thanks, Andrey! Can we update the download page on the website?
>> > >> >
>> > >> > D.
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Ability to check and completely fill transactions on creation

[Dmitriy Setrakyan]

Anton,

We cannot have TransactionStartedEvent without having events for all other
transaction states, like TransactionPreparedEvent,
TransactionCommittedEvent, etc. Considering this, I sill do not like the
design, as we would have to create many extra event classes.

Instead, I would suggest that you create TransactionStateChangeEvent, which
would have previous and new transaction state and would cover all state
changes, not just the start of the transaction. This will make the design
consistent and thorough.

D.

On Tue, May 29, 2018 at 5:39 AM, Anton Vinogradov  wrote:

> Dmitriy,
> I fixed design according to your and Yakov's comments, thanks again for
> clear explanation.
>
> >> 1. You use internal API in public event, i.e. you cannot have user
> >> accessing to IgniteInternalTx instance through TxEvent.
>
> Event definition changed to
> public class TransactionStartedEvent extends EventAdapter {
> private IgniteTransactions tx;
> ...
> }
>
> Not it's 100% public.
>
> >> 2. Throwing runtime errors from listener is not documented and I doubt
> if
> >> it can be fully supported in the pattern you use in TxLabelTest. After
> >> looking at the mentioned test user may think that throwing runtime error
> >> when notified on new node join may prohibit new node joining which is
> not
> >> true. Do you have any example in Ignite when throwing exception from
> >> listener is valid and documented.
>
> Test's logic changed to
> ...
> // Label
> IgniteTransactions tx = evt.tx();
>
> if (tx.label() == null)
> tx.tx().rollback();
> ...
> and
> ...
> // Timeout
> Transaction tx = evt.tx().tx();
>
> if (tx.timeout() < 200)
> tx.rollback();
> ...
>
> So, tx will be rollbacked on creation and any commit attempt will cause
> TransactionRollbackException
>
> Full code listing available at
> https://github.com/apache/ignite/pull/4036/files
>
> Dmitriy, Yakov,
> Could you please check and confirm changes?
>
> чт, 24 мая 2018 г. в 16:32, Dmitriy Setrakyan :
>
> > Anton, why do you need to *alter* event sub-system to introduce a new
> > event? Yakov's issue was that you propagated private interface to public
> > API, which is bad of course. Come up with a clean design and it will be
> > accepted.
> >
> > My problem with TransactionValidator is that it only solves a small
> problem
> > for transactions. If we do that, then we will have to add cache
> validators,
> > compute validators, etc, etc, etc. That is why we either should use the
> > existing event subsystem or come up with a holistic design that will work
> > across the whole project.
> >
> > D.
> >
> > On Thu, May 24, 2018 at 1:38 AM, Anton Vinogradov  wrote:
> >
> > > Dmitriy,
> > >
> > > Yakov is against the solution based on event sub-system
> > > >> I think that we should think about some other solution instead of
> > > altering
> > > >> event sub-system.
> > >
> > > Also, I checked is there any chances to fix all the issues found by
> Yakov
> > > and see that solution becomes overcomplicated in that case.
> > > That's why I'm proposing this lightweight solution.
> > >
> > > As for me it's a good idea to have such validator since that's a common
> > > problem at huge deployments when more than one team have access to
> Ignite
> > > cluster and there is no other way to setup tx cretion rules.
> > >
> > > Yakov,
> > >
> > > Could you please share your thoughts on that?
> > >
> > >
> > > чт, 24 мая 2018 г. в 8:58, Dmitriy Setrakyan :
> > >
> > > > On Wed, May 23, 2018 at 4:08 AM, Anton Vinogradov 
> > wrote:
> > > >
> > > > > Dmitriy, Yakov
> > > > >
> > > > > Are there any objections to updated design taking into account the
> > > > comments
> > > > > I provided?
> > > > >
> > > >
> > > > Anton, I do not like an additional validator. I think you can
> > accomplish
> > > > the same with a transaction event. You just need to design it more
> > > cleanly,
> > > > incorporating the feedback from Yakov.
> > > >
> > >
> >
>

Re: WAL iterator unexpected behavior

[Dmitriy Setrakyan]

Dmitriy,

Thanks for initiating the discussion!

I am not sure I understand the issue fully, but in my view we should not
allow iteration through WAL if we cannot deserialize a portion of it. We
must require that ignite-index is in the classpath, so I would just fail
right away with exception.

D.


On Tue, May 29, 2018 at 6:53 AM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Igniters,
>
> I faced a problem with iterate over WAL.
>
> Let's imagine that we write WAL and write some record which depends from
> ignite-index.
> And then move files to another machine, which does not have ignite-index in
> the classpath.
> Now try to iterate over WAL.
>
> If iterator can not deserialize some record he try to go next and
> execute advanceSegment() for switching to the next segment.
> (ignite-index not in the classpath)
>
> 1. Iterator skips part of the WAL because advanceRecord() failed to
> deserialize record.
> 2. Iterator does not signal that there was an error.
>
> In my opinion, it is not that we expected.
>
> Any comments?
>

Re: [RESULT] [VOTE] Apache Ignite 2.5.0 Release (RC1)

[Dmitriy Setrakyan]

On Tue, May 29, 2018 at 9:08 AM, Andrey Gura  wrote:

> RPM and DEB packages are published to bintray [1] [2]
>
> [1] https://bintray.com/apache/ignite-rpm
> [2] https://bintray.com/apache/ignite-deb


Thanks, Andrey! Can we update the download page on the website?

D.

Re: IGNITE-8238 - Operation can fails with unexpected RuntimeException when node is stopping

[Dmitriy Setrakyan]

As suggested before, please do not put blank ticket numbers into subjects
because no one understands ticket numbers. Please add titles or some other
context to the subject. This will improve the level of engagement from the
community.

I have changed the subject of this thread, let's continue the discussion
here.

D.

On Tue, May 29, 2018 at 9:10 AM, Александр Меньшиков 
wrote:

> Hi, Andrew.
>
> You suggested continuing the discussion on dev-list about IGNITE-8238 [1]
> I have reworked my PR [2]. Have I moved in the right direction?
>
> I see 57 usages of `checkpointReadLock` in the core module (without tests).
> And it still unclear for me should I catch all exceptions from
> `checkpointReadLock` or not?
> Some places look okay like usage inside overrides of `GridWorker#body`.
> But for example `GridCacheAdapter#getAllAsync0` makes me worry.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8238
> [2] https://github.com/apache/ignite/pull/3993/files
>

Re: Node.js Thin Client @ npmjs

[Dmitriy Setrakyan]

I generally think that as a community we need to start taking an approach
of separate thin client releases. This goes for Java, .NET, JDBC, ODBC,
Node.JS, etc...

We can still host them in the same Ignite repo, but they should be a
separate download. Of course, they should be included in the overall Ignite
distribution as well.

Node.JS client could be the first one to take this approach. We can then
migrate others as well.

Denis, Pavel, what do you think?

D.

On Thu, May 24, 2018 at 2:11 PM, Pavel Petroshenko 
wrote:

> As Denis said, there is no need to download the entire Ignite repo to
> install the client. Once published the client is going to be installed by
> users with a command:
>
> npm install -g apache-ignite-client
>
> The sources are going to be distributed as a part of the ignite repository,
> yes. But in general, release and installation process for the client and
> the Ignite technically are completely independent.
>
> And moreover: if there is a bug, especially critical, found in the client
> we shouldn't wait for the next Ignite to be released to get it fixed. We
> should be flexible enough to push the fixes and release the clients'
> updates independently at any point in time.
>
> Having an independent release/versioning scheme would allow the clients to
> get bug-fixes (minor version update) or nonbreaking feature-adds or
> improvements (medium version update) between major Ignite releases
> (potentially breaking changes and thus - the major version update). But the
> client and the Ignite versions mapping might be tricky and should be
> clearly documented.
>
> So there are pros and cons.
>
> But I believe the release policy should be consistent across all the Thin
> clients (I'm not talking about the "native" or Thick ones, which heavily
> depend on the Ignite internals and are a different story).
>
> p.
>
>
> On Thu, May 24, 2018 at 1:44 PM, Denis Magda  wrote:
>
> > Once the client is built it will be uploaded to the npmjs repository,
> > right? So, a JS developer can download the client from there without
> > touching the whole Ignite binary release.
> >
> > However, those who download the whole Ignite binary distribution will
> find
> > node.js there (as well as .NET, C++, JDBC and ODBC).
> >
> > --
> > Denis
> >
> > On Thu, May 24, 2018 at 1:06 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Thu, May 24, 2018 at 12:36 PM, Pavel Petroshenko <
> > pa...@petroshenko.com
> > > >
> > > wrote:
> > >
> > > > Fair enough. Consistency with the other clients is a good argument.
> > > >
> > > >
> > > Pavel, I would discuss it a bit more. Does it really make sense for a
> > > node.js user to download the whole Ignite distribution just to get a
> > > node.js client?
> > >
> >
>

Re: Node.js Thin Client @ npmjs

[Dmitriy Setrakyan]

On Thu, May 24, 2018 at 12:36 PM, Pavel Petroshenko 
wrote:

> Fair enough. Consistency with the other clients is a good argument.
>
>
Pavel, I would discuss it a bit more. Does it really make sense for a
node.js user to download the whole Ignite distribution just to get a
node.js client?

Re: Node.js Thin Client @ npmjs

[Dmitriy Setrakyan]

On Thu, May 24, 2018 at 11:42 AM, Igor Sapego  wrote:

> Well, all other clients have the same version as Ignite.
> Are there any reasons to make a Node.js client some
> kind of a special case?
>
>
I may have spoken too soon. Do we plan to include the node.js client into
Ignite build? If yes, then it will be built anew with every Ignite release,
so the same versioning makes sense in this case.

D.

Re: IGNITE-8583 - review needed

[Dmitriy Setrakyan]

I continuously ask not to provide naked ticket numbers in the subject.
Please also add titles or descriptions. I doubt anyone in the community can
tell what this is about just by looking at the number.

D.

On Thu, May 24, 2018 at 4:09 AM, Dmitry Pavlov 
wrote:

> Hi Dmitriy,
>
> According to upsource
> https://reviews.ignite.apache.org/ignite/review/IGNT-CR-623 review was
> already started by Alexey G. and  Alexey K.
>
> Sincerely,
> Dmitriy Pavlov
>
>
>
> чт, 24 мая 2018 г. в 14:04, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com
> >:
>
> > Igniters,
> >
> > I fixed DataStorageMetricsMXBean.getOffHeapSize, please review my
> changes.
> >
> > https://issues.apache.org/jira/browse/IGNITE-8583
> >
>

Re: Node.js Thin Client @ npmjs

[Dmitriy Setrakyan]

On Thu, May 24, 2018 at 10:03 AM, Pavel Petroshenko 
wrote:

> Igor,
>
> Are you proposing to update the Thin Client versions on every Ignite
> release regardless of the changes made to them?
>
> I tend to think that an independent versioning scheme for the Thin Clients
> might be more flexible/meaningful.
>

Pavel, I agree, the thin clients should have independent versioning.

Re: Ability to check and completely fill transactions on creation

[Dmitriy Setrakyan]

Anton, why do you need to *alter* event sub-system to introduce a new
event? Yakov's issue was that you propagated private interface to public
API, which is bad of course. Come up with a clean design and it will be
accepted.

My problem with TransactionValidator is that it only solves a small problem
for transactions. If we do that, then we will have to add cache validators,
compute validators, etc, etc, etc. That is why we either should use the
existing event subsystem or come up with a holistic design that will work
across the whole project.

D.

On Thu, May 24, 2018 at 1:38 AM, Anton Vinogradov  wrote:

> Dmitriy,
>
> Yakov is against the solution based on event sub-system
> >> I think that we should think about some other solution instead of
> altering
> >> event sub-system.
>
> Also, I checked is there any chances to fix all the issues found by Yakov
> and see that solution becomes overcomplicated in that case.
> That's why I'm proposing this lightweight solution.
>
> As for me it's a good idea to have such validator since that's a common
> problem at huge deployments when more than one team have access to Ignite
> cluster and there is no other way to setup tx cretion rules.
>
> Yakov,
>
> Could you please share your thoughts on that?
>
>
> чт, 24 мая 2018 г. в 8:58, Dmitriy Setrakyan :
>
> > On Wed, May 23, 2018 at 4:08 AM, Anton Vinogradov  wrote:
> >
> > > Dmitriy, Yakov
> > >
> > > Are there any objections to updated design taking into account the
> > comments
> > > I provided?
> > >
> >
> > Anton, I do not like an additional validator. I think you can accomplish
> > the same with a transaction event. You just need to design it more
> cleanly,
> > incorporating the feedback from Yakov.
> >
>

Re: Ability to check and completely fill transactions on creation

[Dmitriy Setrakyan]

On Wed, May 23, 2018 at 4:08 AM, Anton Vinogradov  wrote:

> Dmitriy, Yakov
>
> Are there any objections to updated design taking into account the comments
> I provided?
>

Anton, I do not like an additional validator. I think you can accomplish
the same with a transaction event. You just need to design it more cleanly,
incorporating the feedback from Yakov.

Re: IEP-4, Phase 2. Using BL(A)T for in-memory caches.

[Dmitriy Setrakyan]

Ed,

Anyone speaking Russian would agree that BLAT is not a good name :) Let's
steak to BLT.

D.

On Wed, May 23, 2018 at 5:34 AM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Igniters,
>
> We have invested too much in explaining BLAT. So, it would hard to change
> the name.
> I.e. I propose to save this term.
>
>
> New names for auto-adjust control.
>
> 1. org.apache.ignite.IgniteCluster
>
> *Add*
> isBaselineAutoAdjustEnabled()
> setBaselineAutoAdjustEnabled(boolean enabled);
> setBaselineAutoAdjustTimeout(long timeoutInMs);
> setBaselineAutoAdjustMaxTimeout(long timeoutInMs);
>
> 2. org.apache.ignite.configuration.IgniteConfiguration
>
> *Add*
> IgniteConfiguration setBaselineAutoAdjustEnabled(boolean enabled);
> IgniteConfiguration setBaselineAutoAdjustTimeout(long timeoutInMs);
> IgniteConfiguration setBaselineAutoAdjustMaxTimeout(long timeoutInMs);
>
> Any objections?
>
>
>
> On Fri, May 4, 2018 at 10:01 PM, Dmitriy Setrakyan 
> wrote:
>
> > I do not like the name "current" on the methods. I think we should just
> > remove it, e.g. currentAffinityTopology() -> affinityTopology()
> >
> > D.
> >
> > On Fri, May 4, 2018 at 6:17 AM, Eduard Shangareev <
> > eduard.shangar...@gmail.com> wrote:
> >
> > > Igniters,
> > >
> > > With Vladimir's help, we analyzed another solution's approaches.
> > > And decided to simplify our affinity topology auto-adjusting.
> > >
> > > It should be enough to be able to turn on/off auto-adjusting (flag) and
> > set
> > > 2 timeouts if it is working:
> > > -soft timeout which would be used if there was no other node
> joins/exits;
> > > -hard timeout which we would track from first discovery event and if it
> > > reached then immediately would change affinity topology.
> > >
> > > All other strategies could be realized with API usage
> > (setAffinityTopology)
> > > and metrics tracking by user's monitoring tools.
> > >
> > > So, I suggest next API changes:
> > >
> > > org.apache.ignite.IgniteCluster
> > >
> > > *Deprecate*:
> > > Collection currentBaselineTopology();
> > > void setBaselineTopology(Collection
> > baselineTop);
> > > void setBaselineTopology(long topVer);
> > >
> > > *Replace them with*
> > > Collection currentAffinityTopology();
> > > void setAffinityTopology(Collection
> > affinityTop);
> > > void setAffinityTopology(long topVer);
> > >
> > > *Add*
> > > isAffinityTopologyAutoAdjustEnabled()
> > > setAffinityTopologyAutoAdjustEnabled(boolean enabled);
> > >
> > > org.apache.ignite.configuration.IgniteConfiguration
> > >
> > > *Add*
> > > IgniteConfiguration setAffinityTopologyAutoAdjustEnabled(boolean
> > enabled);
> > > IgniteConfiguration setAffinityTopologyAutoAdjustTimeout(long
> > > timeoutInMs);
> > > IgniteConfiguration setAffinityTopologyAutoAdjustMaxTimeout(long
> > > timeoutInMs);
> > >
> > >
> > > An open question is could we rename or duplicate BaselineNode with
> > > AffinityNode?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Apr 27, 2018 at 6:56 PM, Ivan Rakov 
> > wrote:
> > >
> > > > Eduard,
> > > >
> > > > +1 to your proposed API for configuring Affinity Topology change
> > > policies.
> > > > Obviously we should use "auto" as default behavior. I believe,
> > automatic
> > > > rebalancing is expected and more convenient for majority of users.
> > > >
> > > > Best Regards,
> > > > Ivan Rakov
> > > >
> > > >
> > > > On 26.04.2018 19:27, Eduard Shangareev wrote:
> > > >
> > > >> Igniters,
> > > >>
> > > >> Ok, I want to share my thoughts about "affinity topology (AT)
> changing
> > > >> policies".
> > > >>
> > > >>
> > > >> There would be three major option:
> > > >> -auto;
> > > >> -manual;
> > > >> -custom.
> > > >>
> > > >> 1. Automatic change.
> > > >> A user could set timeouts for:
> > > >> a. change AT on any topology change after some timeout
> > > (setATChangeTimeout
> > > >> in seconds);
> > > >> b. change AT o

Re: Upgrading Ignite to JCache 1.1

[Dmitriy Setrakyan]

Igniters,

It will be great if someone in the community would pick this up. The amount
of changes are minimal and many of them only have to do with clarifying the
documentation. However, removing JSR 107 license confusion in 1.1 would be
great for Ignite.

D.

On Tue, May 22, 2018 at 3:04 PM, Denis Magda  wrote:

> Here is a list of all changes:
> https://groups.google.com/forum/#!topic/jsr107/BC1qKqknzKU
>
> The primary argument for the migration is a license. JCache 1.0 is licensed
> by Oracle that causes legal issues for some of the users. Once we upgrade
> to JCache 1.1 the won't longer be a big deal.
>
> However, once we move to 1.1 we need to be sure that we comply with the
> updated specification.
>
> --
> Denis
>
> On Tue, May 22, 2018 at 5:20 AM, Dmitry Pavlov 
> wrote:
>
> > Hi Denis,
> >
> > What was improved in JCache 1.1?
> >
> > Would it be useful for product to change supported spec. version?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 21 мая 2018 г. в 20:12, Denis Magda :
> >
> > > Igniters,
> > >
> > > Eventually, JCache was relicensed to Apache 2.0 and released 1.1
> version:
> > > https://groups.google.com/forum/#!topic/jsr107/BC1qKqknzKU
> > >
> > > Is there anyone interested in upgrading Ignite to the new version for
> the
> > > next release?
> > > https://issues.apache.org/jira/browse/IGNITE-8548
> > >
> > > --
> > > Denis
> > >
> >
>

Re: Ability to check and completely fill transactions on creation

[Dmitriy Setrakyan]

Anton,

The change looks very questionable. We cannot be adding configuration
validators for every piece of Ignite API. What is it you are trying to
achieve?

D.

On Mon, May 21, 2018 at 7:22 AM, Anton Vinogradov  wrote:

> Yakov, thank's for deep check.
>
> >> I think that we should think about some other solution instead of
> altering
> >> event sub-system.
>
> Thank's to your comments now I see that solution is not perfect.
>
> How about to create
>
> interface TransactionsValidator {
>boolean validate(IgniteTransactions tx){
>   ...
>}
> }
>
> and add it's setter to IgniteConfiguration?
>
> icfg.setTransactionsValidator(new CustomTransactionsValidator(...));
>
> In that case we'll gain easy and proper solution allows to check
> transaction configuration before real tx creation.
>
> It will be necessary to add some getters to IgniteTransactions:
> - label()
> - timeout()
> - concurrency() (optional)
> - isolation() (optional)
> - txSize() (optional)
>
> Thoughts?
>
> пн, 21 мая 2018 г. в 16:31, Yakov Zhdanov :
>
> > Ok, then it probably might have been created before this PR. Anyway, I
> > would not bother too much about pt 3.
> >
> > --Yakov
> >
> > 2018-05-21 16:15 GMT+03:00 Dmitry Pavlov :
> >
> > > Hi Yakov,
> > >
> > > I've checked this code and IgniteCacheTestSuite6 includes TxLabelTest,
> so
> > > point 3 can be considered as solved.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 21 мая 2018 г. в 16:05, Yakov Zhdanov :
> > >
> > > > Anton, I have objections. Please do not merge unless we agree on
> > details.
> > > >
> > > > 1. You use internal API in public event, i.e. you cannot have user
> > > > accessing to IgniteInternalTx instance through TxEvent.
> > > > 2. Throwing runtime errors from listener is not documented and I
> doubt
> > if
> > > > it can be fully supported in the pattern you use in TxLabelTest.
> After
> > > > looking at the mentioned test user may think that throwing runtime
> > error
> > > > when notified on new node join may prohibit new node joining which is
> > not
> > > > true. Do you have any example in Ignite when throwing exception from
> > > > listener is valid and documented.
> > > > 3. TxLabelTest is not included into any suite.
> > > > 4. EVT_TX_STARTED - does not seems to be a proper and descriptive
> name
> > > >
> > > > I think that we should think about some other solution instead of
> > > altering
> > > > event sub-system.
> > > >
> > > > Also I want to ask everyone in community to request explicit approval
> > > from
> > > > community members before changing anything in transactional logic.
> > > >
> > > > Thanks!
> > > >
> > > > --Yakov
> > > >
> > >
> >
>

Re: Baseline topology documentation clarified: usage scenarios and definition

[Dmitriy Setrakyan]

Great docs! Any chance we could add some pictures to illustrate the concept
better.

On Fri, May 18, 2018 at 2:15 AM, Denis Magda  wrote:

> Igniters,
>
> With the help of Stanislav Lukyanov and Ivan Rakov, we could make our
> baseline topology documentation much better and vivid. Check up the new
> sections that did a better job explaining the topology, cover common usage
> scenarious and explain how to trigger the rebalancing programmatically if
> it's not expected that the baseline topology's node count to be recovered
> soon:
>
>- https://apacheignite.readme.io/v2.4/docs/cluster-
>activation#section-baseline-topology
>
> 
>- https://apacheignite.readme.io/v2.4/docs/cluster-
>activation#section-usage-scenarios
>
> 
>- https://apacheignite.readme.io/v2.4/docs/cluster-
>activation#section-triggering-rebalancing-programmatically
>
> 
>
> --
> Denis
>

Re: supporting different configuration format json,yaml...

[Dmitriy Setrakyan]

-
> > > > > > > > Ilya Kasnacheev
> > > > > > > >
> > > > > > > > 2018-05-15 16:09 GMT+03:00 Pavel Kovalenko <
> jokse...@gmail.com
> > >:
> > > > > > > >
> > > > > > > > > +1 to Dmitriy G. proposal.
> > > > > > > > >
> > > > > > > > > Since we're moving Ignite towards outside of Java world, we
> > > > should
> > > > > > > > > definitely care about config usability for users who are
> not
> > > > > familiar
> > > > > > > > with
> > > > > > > > > Java/Spring.
> > > > > > > > > If we take a look at any of our XML-configs, we can see a
> lot
> > > of
> > > > > > > > > boilerplate like "", "" -
> > > terms
> > > > > > which
> > > > > > > > say
> > > > > > > > > nothing to users outside of Java world.
> > > > > > > > > When I see such configs my eyes are filled with bloody
> tears.
> > > > > > > > >
> > > > > > > > > I think we should really consider YAML as our additional
> > > approach
> > > > > to
> > > > > > > > > configure Ignite with full replacement instead of XML in
> > > future.
> > > > > > > > > Comparing to XML, YAML is significantly more human-readable
> > and
> > > > > > > > lightweight
> > > > > > > > > format and has stable Java library to parse and translate
> > > config
> > > > > > files
> > > > > > > to
> > > > > > > > > Java objects without extra-magic.
> > > > > > > > >
> > > > > > > > > We can find a lot of famous projects which are using YAML:
> > > Apache
> > > > > > > Flink,
> > > > > > > > > Apache Storm/Heron and one of the our main rivals - Apache
> > > > > Cassandra.
> > > > > > > > >
> > > > > > > > > Some of the projects use simple = config
> form
> > > > > (Kafka,
> > > > > > > > > Spark), some of the projects use their own YAML-like format
> > > > > > (Aerospike,
> > > > > > > > > Tarantool), but it's really difficult to find such project
> > > which
> > > > > has
> > > > > > so
> > > > > > > > > heavy config as us (maybe VoltDB).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2018-05-15 14:02 GMT+03:00 Andrey Gura :
> > > > > > > > >
> > > > > > > > > > Actually sometimes users ask about JSON configuration
> (e.g.
> > > was
> > > > > PR
> > > > > > in
> > > > > > > > > > vertx-ignite project). But it's non trivial task because
> it
> > > > will
> > > > > > > > > > require development of some DSL (or set of DSL's) and
> will
> > > make
> > > > > > > adding
> > > > > > > > > > new configuration elements some kind of pain while we
> > should
> > > be
> > > > > > > > > > focused on basic functionality: data grid, persistence,
> > etc.
> > > > > > > > > >
> > > > > > > > > > I just believe that configuration format is not so
> > important
> > > > > aspect
> > > > > > > > > > and this task is out of product scope.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, May 15, 2018 at 12:56 PM, Dmitriy Setrakyan
> > > > > > > > > >  wrote:
> > > > > > > > > > > I still do not understand *why* do we need to add
> > > additional
> > > > > > > formats
> > > > > > > &

Re: SqlLine script and Java9

[Dmitriy Setrakyan]

Petr, do we really need to check Java version for SqlLine at all? I think
it will work with any version of Java, so restricting it to a certain
version is just wrong.

What do you think about removing the Java version check altogether from the
SqlLine script?

D.


On Tue, May 15, 2018 at 10:21 AM, Petr Ivanov  wrote:

> Not yet.
>
> After 2.5 release, I plan to prepare a roadmap for Apache Ignite support
> of future Java releases (based on their’s current release cycle) and to
> coordinate corresponding works for introducing support of future Java
> versions.
>
>
>
> > On 15 May 2018, at 01:55, Dmitriy Setrakyan 
> wrote:
> >
> > Thanks Petr,
> >
> > How about Java 10? Do we support it for SqlLine as well?
> >
> > D.
> >
> > On Mon, May 14, 2018 at 1:53 PM, Petr Ivanov 
> wrote:
> >
> >> Tested, review and merged to master and ignite-2.5 branches (thanks to
> >> Andrey Gura for operativeness).
> >>
> >>
> >>
> >>> On 14 May 2018, at 11:40, Petr Ivanov  wrote:
> >>>
> >>> Prepared patch [1], review and testing is required.
> >>>
> >>>
> >>> [1] https://github.com/apache/ignite/pull/3989 <
> >> https://github.com/apache/ignite/pull/3989>
> >>>
> >>>
> >>>
> >>>> On 14 May 2018, at 10:35, Petr Ivanov  >> mr.wei...@gmail.com>> wrote:
> >>>>
> >>>> Filed the ticket [1], will fix.
> >>>>
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-8478 <
> >> https://issues.apache.org/jira/browse/IGNITE-8478>
> >>>>
> >>>>> On 12 May 2018, at 20:10, Dmitriy Setrakyan  >> <mailto:dsetrak...@apache.org>> wrote:
> >>>>>
> >>>>> Igniters,
> >>>>>
> >>>>> Any reason SqlLine tool shipped with Ignite does not work with Java
> 9?
> >> This
> >>>>> is the error I got.
> >>>>>
> >>>>>
> >>>>>> *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
> >>>>>> incorrect.*
> >>>>>> *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK
> >> 1.8.**You
> >>>>>> can also download latest JDK at http://java.com/download <
> >> http://java.com/download>
> >>>>>> <http://java.com/download <http://java.com/download>>.*
> >>>>>
> >>>>>
> >>>>> D.
> >>>>
> >>>
> >>
> >>
>
>

Re: supporting different configuration format json,yaml...

[Dmitriy Setrakyan]

I still do not understand *why* do we need to add additional formats for
the configuration. Can you please show me some users on the user@ list or
stack overflow who asked for it? I just want to make sure that if we are
creating work for ourselves, then someone actually needs it.

D.

On Tue, May 15, 2018 at 12:41 PM, Igor Sapego  wrote:

> I don't think we need to add new formats on server side as there may
> be a lot of different formats for different clients. On the other hand,
> supporting additional formats may be non trivial and error-prone, while
> adding little to a user experience.
>
> For clients, I see no problem in adding for example JSON -> XML
> converter on client side, if JS folks need it.
>
> For servers, adding another configuration format just to make it more
> familiar to users with no Java background seems unreasonable to me
> as well, as there still going to be Java class names in configuration
> and Spring approach in general.
>
> What will change is a XML formatting is going to change to JSON
> formatting, which has no much sense to me, as everyone know XML.
> It is Spring approach what can be confusing to non-Java users, and
> it is not going to change regardless of format.
>
> Best Regards,
> Igor
>
> On Tue, May 15, 2018 at 12:15 PM, Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com> wrote:
>
> > Folks,
> >
> > I guess when work on a thin client will be completed, we get more
> newcomers
> > who use go/python/php/js.
> > And we can do ignite more friendly for them, support familiar formats for
> > configuration.
> >
> > On Tue, May 15, 2018 at 12:13 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Hi Igniters,
> > >
> > > In general I aggree with adding new format, e.g. JSON is more popular
> > than
> > > XML for new applications.
> > >
> > > In the same time I've never heard that user asked this in the user
> list.
> > Or
> > > did I missed such topics?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 15 мая 2018 г. в 9:31, Pavel Tupitsyn :
> > >
> > > > Dmitriy,
> > > >
> > > > We don't need to support different config formats on server in order
> to
> > > add
> > > > that to thin clients.
> > > >
> > > > Thin client protocol provides a way to create a cache with custom
> > config
> > > > [1].
> > > > It is up to thin client library authors to use any config format they
> > > like
> > > > and then convert it into protocol-defined format.
> > > >
> > > > C# thin client uses custom format, for example, not Spring.
> > > >
> > > > [1]
> > > >
> > > > https://apacheignite.readme.io/docs/binary-client-
> > > protocol-cache-configuration-operations#section-op_cache_
> > > create_with_configuration
> > > >
> > > > On Mon, May 14, 2018 at 7:54 PM, Ivan Rakov 
> > > wrote:
> > > >
> > > > > Dmitry,
> > > > >
> > > > > We rely on Spring Framework when we start Ignite node from XML
> > > > > configuration. Spring doesn't easily support another formats of
> > > > > configuration files. I think, the main reason for this is built-in
> > > > ability
> > > > > to validate configuration via XML Schema. We can surely hack this
> > > around
> > > > (I
> > > > > bet there are existing libraries for configuring Spring with JSON),
> > > but I
> > > > > don't think that anyone suffered from inability to statically
> > configure
> > > > > Ignite with json/yaml.
> > > > >
> > > > > Regarding thin clients: makes sense. I suppose necessary mappings
> > will
> > > be
> > > > > implemented as a part of thin client.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > >
> > > > > On 14.05.2018 18:58, Dmitriy Govorukhin wrote:
> > > > >
> > > > >> Hi, Igniters!
> > > > >>
> > > > >> As far as I know, many people work on a thin client for different
> > > > language
> > > > >> (go,js,php...).
> > > > >> Are there any reasons why ignite does not support yaml or json
> > format
> > > > for
> > > > >> configuration? or some other popular format?
> > > > >> In future, it can help to integrate with thin clients, for
> example,
> > js
> > > > >> client may want to dynamic cache start, he passes cache
> > configuration
> > > > (in
> > > > >> native format, for js it will json) through TCP, Ignite node
> unwrap
> > > and
> > > > >> remap to java representation and dynamic start cache.
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: PHP/Python versions for Thin Clients

[Dmitriy Setrakyan]

Thanks Pavel, this makes sense now.

On Tue, May 15, 2018 at 5:30 AM, Pavel Petroshenko 
wrote:

> Hi Dmitriy,
>
> PHP 5.6 and 7.0 are going to be end-of-life shortly [1]. So the minimal
> version for the Thin Client is going to be either 7.1 or 7.2 (I would
> finalize this along with the PHP Thin Client API proposal).
>
> As for Python, there is still some legacy code on 2.7, the oldest active
> 2.x version. However the use of Python 2 is declining as it’s not actively
> developed, doesn’t get new features, and its maintenance is going to be
> stopped in 2020 [2]. Python 3 is a strong leader with 75% and Python 2 is
> used as the main interpreter by only 25% (rapidly declining) [3]. So I'm
> leaning towards supporting 3.4+ (the oldest active 3.x version). However, I
> would keep the 2.7 in mind for API design.
>
> I hope it makes sense.
>
> Thanks,
> p.
>
> [1] http://php.net/supported-versions.php
> [2] https://legacy.python.org/dev/peps/pep-0373/
> [3] https://www.jetbrains.com/research/python-developers-survey-2017/
>
>
> On Sat, May 12, 2018 at 6:21 AM, Dmitriy Setrakyan 
> wrote:
>
> > Pavel,
> >
> > Can you suggest what would be the advantages and disadvantages of
> > supporting different versions?
> >
> > D.
> >
> > On Sat, May 12, 2018 at 7:21 AM, Pavel Petroshenko <
> pa...@petroshenko.com>
> > wrote:
> >
> > > Igniters,
> > >
> > > Are there any strong opinions on which language versions should the
> Thin
> > > Clients written in Python and PHP support? Any objections to using PHP
> > 7.1+
> > > and Python 3.5+?
> > >
> > > Thanks,
> > >
> > > p.
> > >
> >
>

Re: SqlLine script and Java9

[Dmitriy Setrakyan]

Thanks Petr,

How about Java 10? Do we support it for SqlLine as well?

D.

On Mon, May 14, 2018 at 1:53 PM, Petr Ivanov  wrote:

> Tested, review and merged to master and ignite-2.5 branches (thanks to
> Andrey Gura for operativeness).
>
>
>
> > On 14 May 2018, at 11:40, Petr Ivanov  wrote:
> >
> > Prepared patch [1], review and testing is required.
> >
> >
> > [1] https://github.com/apache/ignite/pull/3989 <
> https://github.com/apache/ignite/pull/3989>
> >
> >
> >
> >> On 14 May 2018, at 10:35, Petr Ivanov  mr.wei...@gmail.com>> wrote:
> >>
> >> Filed the ticket [1], will fix.
> >>
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-8478 <
> https://issues.apache.org/jira/browse/IGNITE-8478>
> >>
> >>> On 12 May 2018, at 20:10, Dmitriy Setrakyan  <mailto:dsetrak...@apache.org>> wrote:
> >>>
> >>> Igniters,
> >>>
> >>> Any reason SqlLine tool shipped with Ignite does not work with Java 9?
> This
> >>> is the error I got.
> >>>
> >>>
> >>>> *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
> >>>> incorrect.*
> >>>> *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK
> 1.8.**You
> >>>> can also download latest JDK at http://java.com/download <
> http://java.com/download>
> >>>> <http://java.com/download <http://java.com/download>>.*
> >>>
> >>>
> >>> D.
> >>
> >
>
>

Re: tracking SQL queries in 2.5

[Dmitriy Setrakyan]

On Mon, May 14, 2018 at 3:55 PM, Vladimir Ozerov 
wrote:

> Hi Dima,
>
> This is not that simple unfortunately. We need to build the whole
> infrastructure for cache-less SQL queries. This is not very complex, but
> require considerable efforts. Let's do that in AI 2.6 scope.
>

Got it, thanks!

SqlLine script and Java9

[Dmitriy Setrakyan]

Igniters,

Any reason SqlLine tool shipped with Ignite does not work with Java 9? This
is the error I got.


> *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
> incorrect.*
> *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 1.8.**You
> can also download latest JDK at http://java.com/download
> .*


D.

Integration with JHipster

[Dmitriy Setrakyan]

Igniters,

Jhipster is a very popular Spring Boot + Angular framework. Many caching
frameworks integrate with it, including EhCache, Hazelcast, and Infinispan:

https://www.jhipster.tech/using-cache/

I think the integration is fairly simple and we should provide our own.

Thoughts?

D.

tracking SQL queries in 2.5

[Dmitriy Setrakyan]

Igniters,

I just noticed this horrible ticket:
https://issues.apache.org/jira/browse/IGNITE-6677

Apparently, a couple of releases back the community added support for
"CREATE TABLE" command and for some reason nobody checked that query
metrics on the created tables do not work.

Do we have any idea of what it would take to fix? If it is a simple fix, I
would like to squeeze it into 2.5 release.

D.

Re: PHP/Python versions for Thin Clients

[Dmitriy Setrakyan]

Pavel,

Can you suggest what would be the advantages and disadvantages of
supporting different versions?

D.

On Sat, May 12, 2018 at 7:21 AM, Pavel Petroshenko 
wrote:

> Igniters,
>
> Are there any strong opinions on which language versions should the Thin
> Clients written in Python and PHP support? Any objections to using PHP 7.1+
> and Python 3.5+?
>
> Thanks,
>
> p.
>

Re: async operation is not fair async

[Dmitriy Setrakyan]

Vladimir,

In general I agree, but I do get greatly *close-minded* (pun intended)
whenever users' code that worked for the past several years all of a sudden
gets deadlocked after an upgrade. Making this feature optional is even
worse and more confusing. In this case the best action is no action at all.

BTW, would be interesting to find out how Oracle async driver behaves in
this case.

D.





On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov 
wrote:

> Guys,
>
> To build a great product we should be open minded and look to the future,
> not to the past.
>
> Dima raised very valid point - why async is not async? Current programming
> culture and demanding performance requirements pushes users towards
> reactive-style programming. I do not want my thread to ever be blocked.
> Instead, I want to send a number of concurrent commands and optionally
> subscribe to final result. So trully async API makes total sense to me.
>
> But personally, my primary interest in this area is SQL. Oracle is
> preparing new async driver. ADBA - async database access. It was presented
> on recent JavaOne [1]. It is under active development right now - juse
> weave through the mailing list [2]. Some prototypes are already there [3].
> PostgreSQL community even started adopted it [4]!
>
> I am not pushing for immediate actions, but at least we should understand
> which way the wind is blowing. As a mid-term goals I would propose to
> finally remove thread ID from our PESSIMISTIC transactions to allow for
> suspend/resume in different threads. And as a next step I would think on
> adopting async cache and SQL APIs.
>
> Vladimir.
>
> [1]
> http://www.oracle.com/technetwork/database/application-development/jdbc/
> con1491-3961036.pdf
> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>
> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan 
> wrote:
>
> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com> wrote:
> >
> > > I will edit IGNITE-8475, and remove all part that belong to the public
> > api.
> > > Is it acceptable for you?
> > >
> >
> > Everything is acceptable, as long as the public API is safe :)
> >
>

Re: async operation is not fair async

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> I will edit IGNITE-8475, and remove all part that belong to the public api.
> Is it acceptable for you?
>

Everything is acceptable, as long as the public API is safe :)

Re: async operation is not fair async

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 7:23 PM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Dmitriy S,
>
> If it will be in the internal package, and only for internal usage, are you
> agree with changes?
>

Yes, but please be careful not to create deadlocks for ourselves.

Can you please close the ticket?

D.

Re: async operation is not fair async

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 6:49 PM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Dmitriy S,
>
> It is not broke existing code, because for use this ability you must use
> decorator "withFairSycn()".
>
> What about the argument of Vladimir?
>

Here is Vladimir's quote:


*This would also be helpful for transactional SQL as it would allow to
hide**network
latency. But there is a problem - deadlocks. We need to inform user*
*that this mode should be used with great care. *


I would rather not change anything instead of increasing the probability of
deadlocks. This was the main reason for the current behavior to begin with.

In my view, if something is needed for the transactional SQL, please add it
internally. Let's not corrupt the public API by adding dangerous methods.

D.

Re: async operation is not fair async

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 6:38 PM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Dmitriy S,
>
> Why method named as "async" but does not work as async? This is misleading.
>
> getAllAsync() is a special case. Not always you can use getAllAsync()
> instead
> of multiple getAsync().
> In this topic, I wanna discuss problem not only for the GET operation but
> also all async operation behavior in the one thread.
>
> In compute grid we can run multiple async compute operation in the one
> thread, so why we can not do this for cache?


Because it will break a lot of existing code and create bugs we cannot even
predict at this point. I am not sure why has this become a problem. Is it
preventing us from accomplishing some other task? If not, then I propose to
drop it.

D.

Re: async operation is not fair async

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 6:29 PM, Dmitry Pavlov 
wrote:

> IMO you can complete async operations one before another if these
> operations are related to independent data.
>
> It is strange why Ignite users are not confused by current API. So I
> support Dmitriy's G. suggestion.
>

Again, this is a solution looking for a problem. Nobody complains about
this, so there really isn't any issue. There are so many other tasks we
could focus on. Let's not fix what's not broken.

D.

Re: async operation is not fair async

[Dmitriy Setrakyan]

Guys,

I am not sure I like this approach, especially for this code:

f1=cache.getAsync(key1);
f2=cache.getAsync(key2);

You cannot complete f2 before f1. If you do, the code is unusable and it is
impossible to predict anything. If you need to get 2 elements
asynchronously, use getAllAsync() instead.

Let's not introduce such usability issues into our API and stop trying to
fix what's not broken. I propose to close the ticket as "Won't Fix".

D.

On Fri, May 11, 2018 at 6:14 PM, Dmitriy Govorukhin <
dmitriy.govoruk...@gmail.com> wrote:

> Igniters,
>
> I created the issue. IGNITE-8475
> 
>
> Any comments are welcome.
>
> On Fri, May 11, 2018 at 6:32 PM, Ivan Rakov  wrote:
>
> > Agree. "fair" is more descriptive.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 11.05.2018 18:30, Dmitriy Govorukhin wrote:
> >
> >> Ivan,
> >>
> >> My suggestion "withFairAsync()". What do you think?
> >>
> >> On Fri, May 11, 2018 at 6:23 PM, Ivan Rakov 
> >> wrote:
> >>
> >> I think, the best option from API side is to add decorating
> >>> withExplicitAsync() method.
> >>> We already have withKeepBinary, withExpiryPolicy and so on.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
> >>> On 11.05.2018 18:18, Dmitriy Govorukhin wrote:
> >>>
> >>> Vladimir,
> 
>  Should we create the new cache adapter or rework GridCacheAdapter?
> 
>  On Fri, May 11, 2018 at 5:52 PM, Vladimir Ozerov <
> voze...@gridgain.com>
>  wrote:
> 
>  +1
> 
> > This would also be helpful for transactional SQL as it would allow to
> > hide
> > network latency. But there is a problem - deadlocks. We need to
> inform
> > user
> > that this mode should be used with great care.
> >
> > On Fri, May 11, 2018 at 5:21 PM, Dmitriy Govorukhin <
> > dmitriy.govoruk...@gmail.com> wrote:
> >
> > Hi Igniters,
> >
> >> I have a question. Why our async operation in not really async?
> >>
> >> GridCacheAdapter.syncOp has awaitLastFut(); this call wait last
> async
> >> operation completed.
> >>
> >> This means all async operation in one thread will be executed one by
> >> one
> >> but not in parallel. Async operation is not async.
> >>
> >> Example for atomic cache
> >>
> >> f1=cache.getAsync(key1);
> >> f2=cache.getAsync(key2);
> >>
> >> f1 always will be complete before f2.
> >>
> >> I want to have the ability run multiple async operations in one
> >> thread.
> >> What do you think?
> >>
> >> Maybe we can add new cache adapter with fair async operations?
> >>
> >>
> >>
> >
>

Re: NodeJS thin client: full API

[Dmitriy Setrakyan]

On Fri, May 11, 2018 at 9:14 AM, Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Not yet. Need a help with that.
>

I think we definitely need a load test before we merge to master. Can
anyone in the community assist Alexey?

Re: NodeJS thin client: full API

[Dmitriy Setrakyan]

This is great! Finally a native NodeJS client for Ignite.

Alexey, in addition to the functional tests, were you able to perform any
load tests?

D.

On Fri, May 11, 2018 at 12:07 AM, Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Folks,
>
> The next version is ready -
> in the pull request [1] or directly in the repo [2].
>
> The version includes:
> - full API
> - full implementation
> - examples
> - tests (cover the full API but might need to be updated/extended)
> - docs
>
> The details are in the readme [3]
>
> Regards,
> -Alexey
>
> [1] https://github.com/apache/ignite/pull/3978
> [2] https://github.com/nobitlost/ignite/tree/master/modules/plat
> forms/nodejs
> [3] https://github.com/nobitlost/ignite/blob/master/modules/plat
> forms/nodejs/README.md
>

Re: Using GraalVM instead of standard JVM

[Dmitriy Setrakyan]

Would be nice to have a TC run on Graal, just to have an understanding
whether we support it or not.

D.

On Wed, May 9, 2018 at 4:28 PM, Denis Magda  wrote:

> The performance might become better just by replacing HotSpot with Graal,
> but something suggests me that Ignite has to be adopted for this JVM (as
> well as for Azul VM) to get more benefits. Probably, someone will get
> interested and pick this task up.
>
> What stands out is that the Graal folks also see this VM as an opportunity
> to run custom code on a database side like Oracle or MySQL:
> https://oracle.github.io/oracle-db-mle/ It's a sort of their response to
> compute grid functionality of data grids and Hadoop ecosystem.
>
> --
> Denis
>
> On Wed, May 9, 2018 at 5:23 AM, sbeaupre 
> wrote:
>
> > This is just a thought that came out of a discussion with Dimitry this
> > morning. Recently Oracle has released GraalVM 1.0 after many years of
> > research and development, as a replacement for standard JVM.
> >
> > It should come with huge improvements on several areas (interesting for
> > ignite: AOT, native compilation, remove object allocation in many cases,
> > ...)
> >
> > Any interest from GG in this? Do you guys think it would give ignite a
> > performance boost (haven't tested it myself, just checking if it is
> > worthwhile in the first place, probably low on our prio list).
> >
> > More info:
> > - GraalVM for Java:
> > http://www.graalvm.org/docs/why-graal/#for-java-programs
> > - Twitter is running GraalVM in production for a while now:
> > https://www.youtube.com/watch?v=pR5NDkIZBOA
> > - Getting started:
> > http://www.graalvm.org/docs/getting-started/
> >
> > regards,
> >
> > Sven
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

Re: missing website info

[Dmitriy Setrakyan]

On Tue, May 8, 2018 at 6:23 PM, Denis Magda  wrote:

> Dmitriy,
>
> Presently this information is scattered and presented under sections named
> differently. Agree with the format proposed by you.
> https://issues.apache.org/jira/browse/IGNITE-8455
>
> In addition, we should cover another mode which is indexes in RAM and data
> on disk to get a better technical cost of ownership of Ignite's cluster. I
> heard it's already possible to set up this kind of configuration.
>

Agree, but we do not have this mode yet. When we do, we should definitely
cover it.

Re: IGNITE-3999 review

[Dmitriy Setrakyan]

Igniters, let's make sure we add ticket description to the email subject in
future, so the community will know what the ticket is about. Not everyone
has time to look for the ticket, especially when the link is not even
provided in the email.

For example, the subject of this thread should have been: "IGNITE-3999:
Support case insensitive search in SQL"

D.

On Tue, May 8, 2018 at 2:28 PM, Dmitry Pavlov  wrote:

> Hi Vladimir, thank you for stepping in.
>
> This case confirms the usefulness of the practice of discussing the problem
> at the devlist before implementation.
>
> вт, 8 мая 2018 г. в 10:12, Vladimir Ozerov :
>
> > Hi,
> >
> > I provided my comments in the ticket.
> >
> > On Tue, May 8, 2018 at 5:54 AM, Amir Akhmedov 
> > wrote:
> >
> > > Hi Nikolay, Dmitry Pavlov,
> > > Can you take a look at this PR or maybe you have an idea when it
> probably
> > > can be reviewed?
> > >
> > > Thanks,
> > > Amir
> > >
> > > On Wed, Apr 11, 2018 at 10:57 AM, Nikolay Izhikov  >
> > > wrote:
> > >
> > > > Hello, Amir.
> > > >
> > > > Sorry, but no.
> > > >
> > > > I will take a look in a next few days.
> > > >
> > > >
> > > > В Ср, 11/04/2018 в 14:51 +, Amir Akhmedov пишет:
> > > > > Hi Nikolay,
> > > > >
> > > > > Did you have a chance to check my changes?
> > > > >
> > > > > Thanks,
> > > > > Amir
> > > >
> > >
> >
>

Re: Postpone Apache Ignite 2.5 release to fix baseline topology

[Dmitriy Setrakyan]

Completely support the decision to move any BLT behavior changes to 2.6.
However, in 2.5 we need to add usability log messages, which I believe we
already have.

D.

On Tue, May 8, 2018 at 2:15 PM, Andrey Gura  wrote:

> Igniters,
>
> I believe BLT is serious usability problem but rush isn't good idea
> because can lead to new bugs. As release manager I think that we
> should move BLT fix to Apache Ignite 2.6 release and focus on issues
> included to the AI 2.5 release scope.
> I also want inform you that code freeze is planned for Friday, May 11.
>
> Thanks!
>
> On Sun, Apr 29, 2018 at 8:44 PM, Dmitry Pavlov 
> wrote:
> > Hi Dmitriy,
> >
> > As far as I understand manual activation will not be required for
> in-memory
> > mode (same for persistence). Change means we will change node state from
> > 'joined-inactive' to 'joined-active' according to that user defined in
> > policy (cluster grow policy).
> >
> > Default will be allow to rebalance data to joined node, probably, with
> some
> > delay. This detail will be defenetely discussed at dev list before
> > implementation.
> >
> > Pros: Persistent users will not be facing with disabled rebalancing in
> case
> > of node left - (BL)AT will be changed automatically. This also be handled
> > by cluster shrink policy for both in-memory and durable cases.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 28 апр. 2018 г. в 21:10, Dmitriy Setrakyan :
> >
> >> Can someone explain what is the before and after effect for this change
> >> from the usability standpoint. If we are changing BLT for the in-memory
> >> mode, which is the default, then we must think through all the usability
> >> consequences ahead of time. Otherwise, the perception will be that the
> >> product stopped working because someone did not know to activate the
> >> cluster.
> >>
> >> D.
> >>
> >> On Sat, Apr 28, 2018 at 9:27 AM, Denis Magda  wrote:
> >>
> >> > I'm backing up Vladimir's proposal to fix the behavior in 2.5 if it's
> not
> >> > time-consuming. To give you a bit more context on the subj, here is
> why
> >> we
> >> > should have the fix to be delivered in 2.5:
> >> > http://apache-ignite-users.70518.x6.nabble.com/Problems-
> >> > with-persistence-and-partitioned-cache-in-2-4-0-td21325.html
> >> >
> >> > Frankly, it's not the first time I see similar complaints from those
> who
> >> > are on 2.4.
> >> >
> >> > Alex G., Vovan, how hard is it to fix this?
> >> >
> >> > --
> >> > Denis
> >> >
> >> >
> >> > On Sat, Apr 28, 2018 at 7:56 AM, Vladimir Ozerov <
> voze...@gridgain.com>
> >> > wrote:
> >> >
> >> > > Yakov,
> >> > >
> >> > > Messages would help users understand what is wrong earlier, but will
> >> not
> >> > > protect them from additional maintenance which is required in AI 2.4
> >> and
> >> > is
> >> > > supposed to be removed in next AI releases.
> >> > > Please note that in IEP-4 topic I proposed alternative solution -
> >> release
> >> > > AI 2.5 now, but then release AI 2.6 as soon as BLT is fixed. I.e. it
> >> > would
> >> > > be emergency release.
> >> > >
> >> > > Both approaches works for me, the main goal is to deliver original
> >> > defaults
> >> > > ASAP. However, approach with a single release looks better to me
> >> because
> >> > it
> >> > > will minimize number of migrations for users.
> >> > >
> >> > > Vladimir.
> >> > >
> >> > >
> >> > > On Sat, Apr 28, 2018 at 5:47 PM, Yakov Zhdanov  >
> >> > > wrote:
> >> > >
> >> > > > Guys, how about we release 2.5 in the nearest future after adding
> >> > proper
> >> > > > usability log messages that will explain user what to do and also
> >> > output
> >> > > > link to readme.io with the first BLT related message during node
> >> > uptime.
> >> > > > This should not take much time and we can use the same messages
> when
> >> we
> >> > > > have (BL)AT modes in 2.6. I think that adding messages makes sense
> >> and
> >> > > > should be clear for users which is not the case for 2.4.
> >> > > >
> >> > > > --Yakov
> >> > > >
> >> > >
> >> >
> >>
>

Re: Ticket review checklist

[Dmitriy Setrakyan]

Is this list on the Wiki?

On Mon, May 7, 2018 at 7:26 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> This is the checklist I have at the moment. Please let me know if you have
> any comments on existing items, or want to add or remove anything. It looks
> like we may have not only strict rules, but *nice to have* points here as
> well with help of *MUST*, *SHOULD* and *MAY* words as per RFC2119 [1]. So
> please feel free to suggest optional items as well.
>
> 1) API
> 1.1) API compatibility *MUST* be maintained between minor releases. Do not
> remove existing methods or change their signatures, deprecate them instead
> 1.2) Default behaviour "SHOULD NOT* be changed between minor releases,
> unless absolutely needed. If change is made, it *MUST* be described in
> "Migration Guide"
> 1.3) New operation *MUST* be well-documented in code (javadoc, dotnetdoc):
> documentation must contain method's purpose, description of parameters and
> how their values affect the outcome, description of return value and it's
> default, behavior in negative cases, interaction with other operations and
> components
> 1.4) API parity between Java and .NET platforms *SHOULD* be maintained when
> operation makes sense on both platforms. If method cannot be implemented in
> a platform immediately, new JIRA ticket *MUST* be created and linked to
> current ticket
> 1.5) API parity between thin clients (Java, .NET) *SHOULD* be maintained
> when operation makes sense on several clients. If method cannot be
> implemented in a client immediately, new JIRA ticket *MUST* be created and
> linked to current ticket
> 1.6) All exceptions thrown to a user *MUST* have explanation how to
> resolve, workaround or debug an error
>
> 2) Compatibility
> 2.1) Persistence backward compatibility *MUST* be maintained between minor
> releases. It should be possible to start newer version on data files
> created by the previous version
> 2.2) Thin client forward and backward compatibility *SHOULD* be maintained
> between two consecutive minor releases. If compatibility cannot be
> maintained it *MUST* be described in "Migration Guide"
> 2.3) JDBC and ODBC forward and backward compatibility *SHOULD* be
> maintained between two consecutive minor releases. If compatibility cannot
> be maintained it *MUST* be described in "Migration Guide"
>
> 3) Tests
> 3.1) New functionality *MUST* be covered with unit tests for both positive
> and negative use cases
> 3.2) All test suites *MUST* be run before merge to master..There *MUST* be
> no new test failures
>
> 4) Code style *MUST* be followed as per Ignite's Coding Guidelines
>
> Vladimir.
>
> [1] https://www.ietf.org/rfc/rfc2119.txt
>
> On Fri, May 4, 2018 at 4:33 PM, Vladimir Ozerov 
> wrote:
>
> > Hi Dmitry,
> >
> > Yes, I'll do that in the nearest days.
> >
> > On Wed, Apr 25, 2018 at 8:24 PM, Dmitry Pavlov 
> > wrote:
> >
> >> Igniters, the idea was related to small refactorings co-located with
> main
> >> change.
> >>
> >> Main change itself indicates that existing code did not meet the
> criteria
> >> of practice. Approving of standalone refactorings instead contradicts
> with
> >> principle don't touch if it works. So I still like idea of co-located
> >> changes improving code, javadocs, style, etc.
> >>
> >> But let's not argue about this point now, let's summarize the undisputed
> >> points and add it to the wiki. Vladimir, would you please do it?
> >>
> >>
> >> ср, 25 апр. 2018 г. в 16:42, Nikolay Izhikov :
> >>
> >> > Igniters,
> >> >
> >> > I agree with Vova.
> >> >
> >> > Don't fix if it works!
> >> >
> >> > If you 100% sure then it a useful addition to the product - just make
> a
> >> > separate ticket.
> >> >
> >> > В Ср, 25/04/2018 в 11:44 +0300, Vladimir Ozerov пишет:
> >> > > Guys,
> >> > >
> >> > > The problem with in-place refactorings is that you increase affected
> >> > scope.
> >> > > It is not uncommon to break compatibility or public contracts with
> >> even
> >> > > minor things. E.g. recently we decided drop org.jsr166 package in
> >> favor
> >> > of
> >> > > Java 8 classes. Innocent change. Result - broken storage. Another
> >> problem
> >> > > is conflicts. It is not uncommon to have long-lived branches which
> we
> >> > need
> >> > > to merge with master over and over again. And a lot of refactorings
> >> cause
> >> > > conflicts. It is much easier to resolve them if you know that logic
> >> was
> >> > not
> >> > > affected as opposed to cases when you need to resolve both renames
> and
> >> > > method extractions along with business-logic changes.
> >> > >
> >> > > I'd like to repeat - if you have a time for refactoring then you
> >> > definitely
> >> > > have a time to extract these changes to separate PR and submit a
> >> separate
> >> > > ticket. I am quite understand what "low priority" do you mean if you
> >> do
> >> > > refactorings on your own.
> >> > >
> >> > > On Tue, Apr 24, 2018 at 10:52 PM, Andrey Kuznetsov <
> stku...@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > +1

Re: MTCGA 5th mass run-all

[Dmitriy Setrakyan]

Great progress and thanks to everyone contributing! I can hardly wait for
the 0-failed-tests day. Hope it is coming some time in future :)

On Mon, May 7, 2018 at 3:38 AM, Dmitry Pavlov  wrote:

> Hi Folks,
>
> I've started mass run-all last weekend. In short -  361 tests failed, and
> 12 suites were unable to complete ( vs tests 309 suites 24 - 3 weeks
> early).
>
> We have markable positive dynamics in fixing existing tests. Current failed
> test count grow came from new ZookeeperDiscovery suites (123 from ZK2, 24
> from ZK1).
>
> It means a huge number of old test was fixed, it is approx -95 test
> failures.
>
> It is hard to mention each contributor personally due to a number of
> participants, and I'm glad this activity involves so many people. Thank
> you!
>
> Sincerely,
> Dmitriy Pavlov
>
> I've also attached full results to wiki
> https://cwiki.apache.org/confluence/pages/viewpageattachments.action?
> pageId=73631266&sortBy=date&highlight=Ignite+Teamcity+-+
> current+failures.pdf&
>

Re: memory-only mode for Ignite indexes

[Dmitriy Setrakyan]

On Mon, May 7, 2018 at 2:09 AM, Vladimir Ozerov 
wrote:

> Hi Dima,
>
> Update with indexes would definitely be slower than update without them.
> The question is how much slower. For now the slowdown comes mostly from
> excessive data page reads ([1] and [2] in my previous email) leading to
> page evictions and additional IO. To the contrast, usually only a single
> page write is needed to update an index. Correct index implementation ([1]
> and [2] from previous email) would eliminate data page reads altogether and
> should give dramatic speedup.
>

Sounds great. The changes you are suggesting should give us a great
performance boost. Hopefully they should not take too long to implement.

Regardless, once you are done, we should still perform a benchmark with
persistence indexes vs memory-only indexes. If memory-only will be more
than 20% faster, we should still implement it.

D.

Re: memory-only mode for Ignite indexes

[Dmitriy Setrakyan]

Vladimir, my comments are inline...

On Sat, May 5, 2018 at 6:12 AM, Vladimir Ozerov 
wrote:

> In general I do not support this initiative. There are two serious reasons
> for that:
> 1) Our indexes are slow on updates due to architectural flaws. First, every
> index entry must be of fixed size. For this reason we cannot inline full
> values in general case and suffer from data page lookups [1]. Second, final
> comparisons always compare primary keys, so another lookup is needed [2].
> Third, our indexes are fat because we are lacking prefix compression [3].
>

These all seem like great optimization and we should definitely do them.
However, I am of the strong opinion that even after these optimizations,
the data ingestion speed will be much slower with the persistence turned
on. Am I wrong?


> 2) Some vendors do have memory-only indexes - SQL Server, Couchbase,
> MemSQL, to name a few. But they are memory optimized - no pages, no BTrees.
> Lock-free skiplist is used instead. This is correct design which really
> fast. But we are very far from it at the moment.
>

I have not heard complaints about our BTree indexes being slow in memory. I
only hear complaints about the slow-downs whenever the persistence is
turned on and users are ingesting large amounts of data.


> Taking this in count I would not consider memory-only BTree indexes in the
> nearest future. Instead, we should focus on performance. When mentioned
> things are fixed/implemented, our indexes will be both memory-efficient and
> very fast to update.
>

I would agree with you only if there is no performance boost in the short
term. So far, disabling persistence for indexes seems like a very simple
change, but could render a significant performance boost.


>
> [1]
> https://issues.apache.org/jira/browse/IGNITE-8385
> [2]
> https://issues.apache.org/jira/browse/IGNITE-8384
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-
> IndexPrefixCompression
>
> сб, 5 мая 2018 г. в 3:46, Dmitriy Setrakyan :
>
> > Igniters,
> >
> > One of the main complaints I hear from users is that whenever the
> > persistence is turned on, index creation can really slow down the
> > performance, because of massive amounts of writes to disk. The reason
> > Ignite is writing indexes to disk is to support fast restarts - nothing
> > needs to be rebuilt on startup, and Ignite can become operational right
> > away.
> >
> > However, as far as I can tell, most users care about faster operations
> > after the system is started and much less about the startup speed. What
> if
> > we added a mode where we do not persist indexes at all? This way data
> > ingestion and overall throughput will significantly increase (of course,
> at
> > the cost of startup type getting longer because we have to rebuild the
> > indexes).
> >
> > There are 2 ways to achieve this in Ignite. The simplest way is not mark
> > index pages dirty in memory, so they will never participate in
> > check-pointing process. We also have to make sure that index pages never
> > get evicted form memory. This can be done fairly quickly. The
> disadvantage
> > of this approach is that if indexes fill up most of the memory, then it
> > will be very difficult to find a page to evict, which may hurt the
> > performance.
> >
> > The other way is to have a separate in-memory off-heap region for
> indexes.
> > This region should never be persisted. It maybe somewhat bigger
> > refactoring, as we currently do not separate between index and data
> pages.
> > However, the advantage of this approach is that this region can be
> flushed
> > to disk practically as is during a graceful shutdown of the node, and
> hence
> > shorten the restart time.
> >
> > I think we should start from the 1st approach and then think about the
> 2nd
> > one. What do you think?
> >
> > D.
> >
>

missing website info

[Dmitriy Setrakyan]

I have been going through the Ignite website today, and I have noticed that
nowhere on the website we mention various modes on how Ignite native
persistence can be used:

- no disk, data is in memory-only (potentially over a 3rd-party database)
- disk is a copy of the memory (only for recovery purposes)
- disk is a data storage with memory used as a performant caching layer

I believe we should put it in a table and explain it somewhere (homepage?).

Denis, Prachi, what do you think?

D.

memory-only mode for Ignite indexes

[Dmitriy Setrakyan]

Igniters,

One of the main complaints I hear from users is that whenever the
persistence is turned on, index creation can really slow down the
performance, because of massive amounts of writes to disk. The reason
Ignite is writing indexes to disk is to support fast restarts - nothing
needs to be rebuilt on startup, and Ignite can become operational right
away.

However, as far as I can tell, most users care about faster operations
after the system is started and much less about the startup speed. What if
we added a mode where we do not persist indexes at all? This way data
ingestion and overall throughput will significantly increase (of course, at
the cost of startup type getting longer because we have to rebuild the
indexes).

There are 2 ways to achieve this in Ignite. The simplest way is not mark
index pages dirty in memory, so they will never participate in
check-pointing process. We also have to make sure that index pages never
get evicted form memory. This can be done fairly quickly. The disadvantage
of this approach is that if indexes fill up most of the memory, then it
will be very difficult to find a page to evict, which may hurt the
performance.

The other way is to have a separate in-memory off-heap region for indexes.
This region should never be persisted. It maybe somewhat bigger
refactoring, as we currently do not separate between index and data pages.
However, the advantage of this approach is that this region can be flushed
to disk practically as is during a graceful shutdown of the node, and hence
shorten the restart time.

I think we should start from the 1st approach and then think about the 2nd
one. What do you think?

D.

Re: IEP-4, Phase 2. Using BL(A)T for in-memory caches.

[Dmitriy Setrakyan]

I do not like the name "current" on the methods. I think we should just
remove it, e.g. currentAffinityTopology() -> affinityTopology()

D.

On Fri, May 4, 2018 at 6:17 AM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Igniters,
>
> With Vladimir's help, we analyzed another solution's approaches.
> And decided to simplify our affinity topology auto-adjusting.
>
> It should be enough to be able to turn on/off auto-adjusting (flag) and set
> 2 timeouts if it is working:
> -soft timeout which would be used if there was no other node joins/exits;
> -hard timeout which we would track from first discovery event and if it
> reached then immediately would change affinity topology.
>
> All other strategies could be realized with API usage (setAffinityTopology)
> and metrics tracking by user's monitoring tools.
>
> So, I suggest next API changes:
>
> org.apache.ignite.IgniteCluster
>
> *Deprecate*:
> Collection currentBaselineTopology();
> void setBaselineTopology(Collection baselineTop);
> void setBaselineTopology(long topVer);
>
> *Replace them with*
> Collection currentAffinityTopology();
> void setAffinityTopology(Collection affinityTop);
> void setAffinityTopology(long topVer);
>
> *Add*
> isAffinityTopologyAutoAdjustEnabled()
> setAffinityTopologyAutoAdjustEnabled(boolean enabled);
>
> org.apache.ignite.configuration.IgniteConfiguration
>
> *Add*
> IgniteConfiguration setAffinityTopologyAutoAdjustEnabled(boolean enabled);
> IgniteConfiguration setAffinityTopologyAutoAdjustTimeout(long
> timeoutInMs);
> IgniteConfiguration setAffinityTopologyAutoAdjustMaxTimeout(long
> timeoutInMs);
>
>
> An open question is could we rename or duplicate BaselineNode with
> AffinityNode?
>
>
>
>
>
>
> On Fri, Apr 27, 2018 at 6:56 PM, Ivan Rakov  wrote:
>
> > Eduard,
> >
> > +1 to your proposed API for configuring Affinity Topology change
> policies.
> > Obviously we should use "auto" as default behavior. I believe, automatic
> > rebalancing is expected and more convenient for majority of users.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 26.04.2018 19:27, Eduard Shangareev wrote:
> >
> >> Igniters,
> >>
> >> Ok, I want to share my thoughts about "affinity topology (AT) changing
> >> policies".
> >>
> >>
> >> There would be three major option:
> >> -auto;
> >> -manual;
> >> -custom.
> >>
> >> 1. Automatic change.
> >> A user could set timeouts for:
> >> a. change AT on any topology change after some timeout
> (setATChangeTimeout
> >> in seconds);
> >> b. change AT on node left after some timeout
> (setATChangeOnNodeLeftTimeout
> >> in seconds);
> >> c. change AT on node join after some timeout
> (setATChangeOnNodeJoinTimeout
> >> in seconds).
> >>
> >> b and c are more specific, so they would override a.
> >>
> >> Also, I want to introduce a mechanism of merging AT changes, which would
> >> be
> >> turned on by default.
> >> Other words, if we reached timeout than we would change AT to current
> >> topology, not that one which was on timeout schedule.
> >>
> >> 2. Manual change.
> >>
> >> Current behavior. A user change AT himself by console tools or web
> >> console.
> >>
> >> 3. Custom.
> >>
> >> We would give the option to set own realization of changing policy
> (class
> >> name in config).
> >> We should pass as incoming parameters:
> >> - current topology (collection of cluster nodes);
> >> - current AT (affinity topology);
> >> - map of GroupId to minimal alive backup number;
> >> - list of configuration (1.a, 1.b, 1.c);
> >> - scheduler.
> >>
> >> Plus to these configurations, I propose orthogonal option.
> >> 4. Emergency affinity topology change.
> >> It would change AT even MANUAL option is set if at least one cache group
> >> backup factor goes below  or equal chosen one (by default 0).
> >> So, if we came to situation when after node left there was only primary
> >> partion (without backups) for some cache group we would change AT
> >> immediately.
> >>
> >>
> >> Thank you for your attention.
> >>
> >>
> >> On Thu, Apr 26, 2018 at 6:57 PM, Eduard Shangareev <
> >> eduard.shangar...@gmail.com> wrote:
> >>
> >> Dmitriy,
> >>>
> >>> I also think that we should think about 2.6 as the target.
> >>>
> >>>
> >>> On Thu, Apr 26, 2018 at 3:27 PM, Alexey Goncharuk <
> >>> alexey.goncha...@gmail.com> wrote:
> >>>
> >>> Dmitriy,
> 
>  I doubt we will be able to fit this in 2.5 given that we did not even
>  agree
>  on the policy interface. Forcing in-memory caches to use baseline
>  topology
>  will be an easy technical fix, however, we will need to update and
>  probably
>  fix lots of failover tests, add new ones.
> 
>  I think it makes sense to target this change to 2.6.
> 
>  2018-04-25 22:25 GMT+03:00 Ilya Lantukh :
> 
>  Eduard,
> >
> > I'm not sure I understand what you mean by "policy". Is it an
> interface
> > that will have a few default implementations and user will be able to
> > create his own one? If so, could you please wr

Re: Greetings

[Dmitriy Setrakyan]

Hi Monil,

Welcome to the Ignite community! Please use dev@ alias going forward
instead of dev-owner@.

Ignite comes with lots of examples. You can also find documentation and
screencasts on the Ignite website. If you run into any questions, please
send them here.

D.

On Tue, May 1, 2018 at 7:44 PM, Monil Shah 
wrote:

> Hello,
>
> My self Monil Shah. I am a software engineer working in the industry for
> past 6 years. I have mostly worked on Java Technologies.
>
> I am really excited to be the part of Apache Ignite.
>
> I have started reading Ignite book. Any documentation on the Ignite
> workings would be appreciated.
>
> I am looking forward to contribute something meaningful to the community.
>
> --
> Thanks and Regards
> Monil Shah
> Software Engineer
> +16018131329
>

Re: cache store usability

[Dmitriy Setrakyan]

On Wed, May 2, 2018 at 4:15 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Vyacheslav,
>
> There is already a warning for this:
> https://github.com/apache/ignite/blob/master/modules/
> core/src/main/java/org/apache/ignite/internal/processors/cache/store/
> GridCacheStoreManagerAdapter.java#L225


Val, why not enable write through if either one is set to true?

cache store usability

[Dmitriy Setrakyan]

Igniters,

This is another usability issue that can be addressed quickly. Apparently,
setWriteBehindEnabled(true) is not enough to enable CacheStore, the
setWriteThrough(true) also needs to be enabled.

https://stackoverflow.com/questions/50118842/write-behind-and-write-through

Why not make it easier for our users? If there are no objections, I would
like to fie a ticket to enable the CacheStore if
setWriteBehindEnabled(true) was turned on.

D.

Re: Contributing

[Dmitriy Setrakyan]

Hi Mihkel,

I just added you to the Ignite contributors list in Jira. You should be
able to assign the ticket to yourself now.

D.

On Mon, Apr 30, 2018 at 6:42 AM, Mihkel Jõhvik 
wrote:

> Hi
>
> I'd like to be added to the Apache Ignite contributors list to work on
> https://issues.apache.org/jira/browse/IGNITE-8293.
>
> Mihkel
>

Re: Postpone Apache Ignite 2.5 release to fix baseline topology

[Dmitriy Setrakyan]

Can someone explain what is the before and after effect for this change
from the usability standpoint. If we are changing BLT for the in-memory
mode, which is the default, then we must think through all the usability
consequences ahead of time. Otherwise, the perception will be that the
product stopped working because someone did not know to activate the
cluster.

D.

On Sat, Apr 28, 2018 at 9:27 AM, Denis Magda  wrote:

> I'm backing up Vladimir's proposal to fix the behavior in 2.5 if it's not
> time-consuming. To give you a bit more context on the subj, here is why we
> should have the fix to be delivered in 2.5:
> http://apache-ignite-users.70518.x6.nabble.com/Problems-
> with-persistence-and-partitioned-cache-in-2-4-0-td21325.html
>
> Frankly, it's not the first time I see similar complaints from those who
> are on 2.4.
>
> Alex G., Vovan, how hard is it to fix this?
>
> --
> Denis
>
>
> On Sat, Apr 28, 2018 at 7:56 AM, Vladimir Ozerov 
> wrote:
>
> > Yakov,
> >
> > Messages would help users understand what is wrong earlier, but will not
> > protect them from additional maintenance which is required in AI 2.4 and
> is
> > supposed to be removed in next AI releases.
> > Please note that in IEP-4 topic I proposed alternative solution - release
> > AI 2.5 now, but then release AI 2.6 as soon as BLT is fixed. I.e. it
> would
> > be emergency release.
> >
> > Both approaches works for me, the main goal is to deliver original
> defaults
> > ASAP. However, approach with a single release looks better to me because
> it
> > will minimize number of migrations for users.
> >
> > Vladimir.
> >
> >
> > On Sat, Apr 28, 2018 at 5:47 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Guys, how about we release 2.5 in the nearest future after adding
> proper
> > > usability log messages that will explain user what to do and also
> output
> > > link to readme.io with the first BLT related message during node
> uptime.
> > > This should not take much time and we can use the same messages when we
> > > have (BL)AT modes in 2.6. I think that adding messages makes sense and
> > > should be clear for users which is not the case for 2.4.
> > >
> > > --Yakov
> > >
> >
>

Re: Apache Ignite 2.5 release

[Dmitriy Setrakyan]

On Fri, Apr 27, 2018 at 6:38 AM, Ivan Rakov  wrote:

> Folks,
>
> I have a fix for a critical issue related to WAL compaction:
> https://issues.apache.org/jira/browse/IGNITE-8393
> In short, if part of WAL archive is broken, attempt to compress it may
> result in spamming warnings in infinite loop.
> I think it's also worth being included in 2.5 release.
>

Let's include it, especially given that the fix is done.

Ivan, on another note, is WAL compaction described anywhere? What do we do
internally to compact WAL and by what factor are we able to reduce the WAL
file size?

D.

Re: IEP-19: Optimize SQL indexes

[Dmitriy Setrakyan]

On Thu, Apr 26, 2018 at 9:09 PM, Vladimir Ozerov 
wrote:

> It is impossible to estimate what is more critical because it would require
> prototypes for every idea to estimate the impact. Instead, we should start
> working on the simplest things, such as IGINTE-8386 [1] or IGNITE-8384 [2].
> And then gradually swtich to more and more complex changes.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8386
> [2] https://issues.apache.org/jira/browse/IGNITE-8384


Vladimir, there is no way for me to tell how his tickets affect any Ignite
users. What will change for our users.

I completely disagree about  about not prioritizing. We should identify how
critical the issues are for our users and start working on them in that
order.

D.

Re: cache size() calculation for MVCC

[Dmitriy Setrakyan]

On Wed, Apr 25, 2018 at 5:45 PM, Vladimir Ozerov 
wrote:

> Yakov,
>
> Thread-per-partition is hardly applicable for general SQL use case as user
> operates on arbitrary data sets. But in general we may track size deltas
> for partitions on transaction level. If transaction span one or several
> partitions, we may hold this data in a single long or Map. If transaction
> spans a lot of partitions, we may store this data in array.
>
> What do you think?
>

Vova, I think you are suggesting maintaining the size as you go. This does
sound like a right approach, assuming that it does not carry noticeable
overhead.

D.

Re: IEP-4, Phase 2. Using BL(A)T for in-memory caches.

[Dmitriy Setrakyan]

Ed, how difficult is this fix? Are you suggesting it for 2.5?

On Thu, Apr 26, 2018 at 3:10 AM, Eduard Shangareev <
eduard.shangar...@gmail.com> wrote:

> Igniters,
> I have described the issue with current approach in "New definition for
> affinity node (issues with baseline)" topic[1].
>
> Now we have 2 different affinity topology (one for in-memory, another for
> persistent caches).
>
> It causes problems:
> - we lose (in general) co-location between different caches;
> - we can't avoid PME when non-BLAT node joins cluster;
> - implementation should consider 2 different approaches to affinity
> calculation.
>
> So, I suggest unifying behavior of in-memory and persistent caches.
> They should all use BLAT.
>
> Their behaviors were different because we couldn't guarantee the safety of
> in-memory data.
> It should be fixed by a new mechanism of BLAT changing policy which was
> already discussed there - "Triggering rebalancing on timeout or manually if
> the baseline topology is not reassembled" [2].
>
> And we should have a policy by default which similar to current one
> (add nodes, remove nodes automatically but after some reasonable delay
> [seconds]).
>
> After this change, we could stop using the term 'BLAT', Basline and so on.
> Because there would not be an alternative. So, it would be only one
> possible Affinity Topology.
>
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/New-
> definition-for-affinity-node-issues-with-baseline-td29868.html
> [2]
> http://apache-ignite-developers.2346864.n4.nabble.com/
> Triggering-rebalancing-on-timeout-or-manually-if-the-baselin
> e-topology-is-not-reassembled-td29299.html#none
>

Re: IEP-19: Optimize SQL indexes

[Dmitriy Setrakyan]

Thanks, Vladimir!

Looking at this IEP, it is not clear which tickets are more critical than
others. Also, the complexity of each ticket is unknown. Is there a way to
provide this information?

D.

On Wed, Apr 25, 2018 at 10:21 PM, Vladimir Ozerov 
wrote:

> Igniters,
>
> I heard a lot of complains around performance of our SQL indexes. Notably -
> slow updates and slow execution of CREATE INDEX command on large data sets.
> I summarized all known possible optimizations under a single IEP [1]. We
> need to start working in this direction, starting from the most simple and
> obvious things.
>
> Please feel free to review IEP tickets and share additional suggestions or
> comments or what else could be done to make our indexes faster.
>
> Vladimir.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 19%3A+SQL+index+update+optimizations
>

Re: cache size() calculation for MVCC

[Dmitriy Setrakyan]

Vova, what about maintaining the size as you go? This way you don't have to
count multiple versions, no?

D.

On Wed, Apr 25, 2018, 5:07 PM Vladimir Ozerov  wrote:

> This is interesting question. Full-scan size may be tremendously slow
> operation on large data sets. On the other hand, printing total number of
> tuples including old and aborted versions make little to no sense as well.
> Looks like we need to choose lesser of two evils. What if we do the
> following:
> 1) Left default behavior as is - O(1) complexity, but includes invalid
> versions
> 2) As Sergey suggested, add new peek mode "MVCC_ALIVE_ONLY" which will
> perform full scan.
>
> Alternatively we may throw an "UnsupportedOperationException" from this
> method - why not?
>
> Thoughts?
>
> On Tue, Apr 24, 2018 at 4:28 PM, Sergey Kalashnikov  >
> wrote:
>
> > Hi Igniters,
> >
> > I need your advice on a task at hand.
> >
> > Currently cache API size() is a constant time operation, since the
> > number of entries is maintained as a separate counter.
> > However, for MVCC-enabled cache there can be multiple versions of the
> > same entry.
> > In order to calculate the size we need to obtain a MVCC snapshot and
> > then iterate over data pages filtering invisible versions.
> > So, it is impossible to keep the same complexity guarantees.
> >
> > My current implementation internally switches to "full-scan" approach
> > if cache in question is a MVCC-enabled cache.
> > It happens unbeknown to users, which may expect lightning-fast
> > response as before.
> > Perhaps, we might add a new constant to CachePeekMode enumeration that
> > is passed to cache size() to make it explicit?
> >
> > The second concern is that cache size calculation is also included
> > into Cache Metrics API and Visor functionality.
> > Will it be OK for metrics and things alike to keep returning raw
> > unfiltered number of entries?
> > Is there any sense in showing raw unfiltered number of entries which
> > may vary greatly from invokation to invokation with just simple
> > updates running in background?
> >
> > Please share your thoughts.
> >
> > Thanks in advance.
> > --
> > Sergey
> >
>

Re: New definition for affinity node (issues with baseline)

[Dmitriy Setrakyan]

On Wed, Apr 25, 2018 at 4:13 AM, Vladimir Ozerov 
wrote:

> Right, as far as I understand we are not arguing on whether BLT is needed
> or not. The main questions are how to properly deliver this feature to
> users and how to deal with co-location issues between persistent and
> non-persistent caches. Looks like change policies are the way to go for the
> first question.
>
> As far as co-location, it is important to note that different affinity
> distribution for in-memory and persistent caches automatically means that
> we loose SQL joins and predictable behavior of any affinity-based
> operations. It means that if we calculated the same affinity for persistent
> and in-memory caches at some point, we cannot re-distribute in-memory
> caches differently if some nodes go down without breaking co-located
> computations, am I right?
>

Vova, you are right, but this is rather an edge case. I doubt there are
many users out there who will need to join memory-only with persistent
caches. What you are describing would be nice to support, but I would not
make it a hard requirement. However, if we choose not to support it, we
should have a very good explanation for why not.

Re: Service grid redesign

[Dmitriy Setrakyan]

On Tue, Apr 24, 2018, 3:59 PM Denis Mekhanikov 
wrote:

> Dmitriy,
>
> After the proposed changes are made the utility cache won't be needed at
> all.
>

I was rather talking about prioritization. In my view, first and foremost
we must fix deployment before anything else.

D.

Re: Apache Ignite 2.4+ Go language client

[Dmitriy Setrakyan]

Any chance we can add key-value support as well?

On Tue, Apr 24, 2018, 2:48 PM Pavel Tupitsyn  wrote:

> Hi Aleksandr,
>
> This is awesome, thank you!
>
> However, let's make it clear that this client supports SQL only,
> and none of the other Thin Client protocol features.
>
> Pavel
>
> On Mon, Apr 23, 2018 at 10:41 PM, Aleksandr Sokolovskii  >
> wrote:
>
> > Hi Oleg,
> >
> > Thanks for your answer.
> >
> > > Community is currently working on formal test specification.
> > Great. Waiting for this one.
> >
> > > As far as NodeJS please note that it is already being developed by
> > community at the moment [1].
> > Cool. I stop my initiatives.
> >
> > Thanks,
> > Aleksandr
> >
> > From: Vladimir Ozerov
> > Sent: 23 апреля 2018 г. 22:35
> > To: dev@ignite.apache.org
> > Subject: Re: Apache Ignite 2.4+ Go language client
> >
> > Hi Alexander,
> >
> > Awesome thing! Please note that before accepting the client we need to
> make
> > sure it is operational. Community is currently working on formal test
> > specification. I hope it will be ready soon.
> >
> > As far as NodeJS please note that it is already being developed by
> > community at the moment [1]. We hope to have it in Apache Ignite 2.6.
> >
> > [1]
> > https://issues.apache.org/jira/browse/IGNITE-
> >
> > пн, 23 апр. 2018 г. в 22:24, Aleksandr Sokolovskii :
> >
> > > Hi All,
> > >
> > > I hope you are well.
> > >
> > > I released Apache Ignite 2.4+ Go language client:
> > > https://github.com/apache-ignite/go-client
> > >
> > > I updated link here:
> > > https://github.com/golang/go/wiki/SQLDrivers
> > >
> > > Is it possible to add link to my repo to this page?:
> > > https://apacheignite.readme.io/docs/binary-client-protocol
> > > or this page:
> > > https://apacheignite-net.readme.io/docs/thin-client
> > > Golang is much more easy to understand than Java or С#.
> > > It’s very easy to pull, build and run test for my library.
> > > I believe it helps another guys to write more thin clients.
> > >
> > > P.S.: I started developing Node.js client also.
> > >
> > > Thanks,
> > > Aleksandr
> > >
> > >
> >
> >
>

Re: Writing a helper for KafkaStreamer

[Dmitriy Setrakyan]

Dmitriy, I think it would make sense to identify a few committers who could
review this ticket and reach out to them.

D.

On Mon, Apr 23, 2018 at 6:05 AM, Dmitry Pavlov 
wrote:

> Hi Igniters,
>
> It seems issue is still in PA state.
>
> Who can pick up https://issues.apache.org/jira/browse/IGNITE-5416 review?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 6 июн. 2017 г. в 10:56, Dmitriy Setrakyan :
>
> > Thanks, Mike! Sounds great.
> >
> > On Tue, Jun 6, 2017 at 12:26 AM, Michael Griggs <
> > michael.gri...@gridgain.com
> > > wrote:
> >
> > > We made a change [1] that required users to re-write code that uses
> > > KafkaStreamer to initialise tuple extractors.  The re-write is not
> > > immediately obvious, and for simple use cases (streaming single tuples)
> > it
> > > is easy to write a helper function that will make the transition
> easier.
> > >
> > >
> > >
> > > I intend to raise a JIRA and submit a PR to implement this.
> > >
> > >
> > >
> > > MG
> > >
> > >
> > >
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-4140
> > >
> > >
> > >
> > > --
> > >
> > > Michael Griggs
> > >
> > > Consultant, EMEA
> > >
> > > GridGain Systems
> > >
> > >
> > >
> > >
> >
>

Re: Replace Cron4J with Quartz for ignite-schedule module.

[Dmitriy Setrakyan]

Dmitriy, who is a good candidate within the community to review this ticket?

On Mon, Apr 23, 2018 at 6:10 AM, Dmitry Pavlov 
wrote:

> Hi Igniters,
>
> it seems ticket https://issues.apache.org/jira/browse/IGNITE-5565 is still
> in PA state. What are our next steps?
>
> Who did review of this patch?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 28 июн. 2017 г. в 1:40, Denis Magda :
>
> > Yakov,
> >
> > No, the mentioned discussion didn’t turn into a JIRA ticket.
> >
> > Alex K., please follow to some thoughts from there and wrap them up in a
> > form of the ticket.
> >
> > —
> > Denis
> >
> > > On Jun 26, 2017, at 2:58 AM, Yakov Zhdanov 
> wrote:
> > >
> > > Guys, I remember we discussed this some time ago.
> > >
> > >
> > http://apache-ignite-developers.2346864.n4.nabble.
> com/Tasks-Scheduling-and-Chaining-td14293.html
> > >
> > > Denis, do you have any ticket or SoW?
> > >
> > > --Yakov
> >
> >
>

Re: write-through when using SQL updates

[Dmitriy Setrakyan]

On Sun, Apr 22, 2018 at 10:16 PM, Denis Magda  wrote:

> If a table is created with CREATE TABLE command then:
>
>- Ignite creates two custom binary types (one for the key and one for
>the value) the columns will be wrapped into. A primary key defines what
>goes into the key. It happens automatically unless...
>- You define names of existing business objects the columns will be
>mapped to. Use KEY_TYPE and VALUE_TYPE parameters for that.
>
> Read more here:
> https://apacheignite-sql.readme.io/docs/create-table#section-description


Finally, I see the documentation. We need a separate section for it with a
proper title. To the least, we will be able to provide a direct link to it
when people ask questions.

D.

Re: write-through when using SQL updates

[Dmitriy Setrakyan]

Denis, I get that we support it, but I still do not understand how. Given
an SQL table in Ignite with several columns, what is the key class and what
is the value class? Where do we document how SQL types map to key-value
types?

D.

On Sun, Apr 22, 2018 at 1:43 PM, Denis Magda  wrote:

> Dmitriy,
>
> According to the docs, INSERTS are transformed into cache.putIfAbsent or
> cache.invokeAll operations:
> https://apacheignite-sql.readme.io/docs/insert#section-description
>
> UPDATES are turned into cache.invokeAll all the times:
> https://apacheignite-sql.readme.io/docs/update#section-description
>
> Once an SQL statement becomes a key-value operation(s) we should update
> both RAM and a 3rd party database following a standard contract:
> https://apacheignite.readme.io/docs/3rd-party-store
>
> SQL experts, please confirm my understanding is correct.
>
> --
> Denis
>
>
> On Sun, Apr 22, 2018 at 7:54 AM, Dmitriy Setrakyan 
> wrote:
>
> > Igniters,
> >
> > Do we support write-through to a 3rd party database when performing SQL
> > inserts and updates? If yes, how does the mapping between SQL and
> > CacheStore key-value API happen?
> >
> > D.
> >
>

write-through when using SQL updates

[Dmitriy Setrakyan]

Igniters,

Do we support write-through to a 3rd party database when performing SQL
inserts and updates? If yes, how does the mapping between SQL and
CacheStore key-value API happen?

D.

Re: Service grid redesign

[Dmitriy Setrakyan]

Thanks! The IEP looks very big.

I would like to remind everyone that one of the biggest problems we have
with services is that it uses a replicated cache internally to do the
deployment. When all nodes have the same service configured in the XML
file, then all nodes will try to initiate a put into the replicated cache
at the same time for the same key, which results in lots of contention and
significantly slows down the startup speed.

In my view, this is what needs to be fixed first. Is there a ticket for it?

D.

On Sat, Apr 21, 2018 at 11:16 AM, Denis Magda  wrote:

> Dmitriy,
>
> Consider IEP page as a summary that was updated along the way:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 17%3A+Oil+Change+in+Service+Grid
>
> As far as I understand, Denis is going to create JIRA tickets basing on the
> discussion results.
>
> --
> Denis
>
> On Sat, Apr 21, 2018 at 3:04 AM, Dmitriy Setrakyan 
> wrote:
>
> > I am not sure why we are discussing a potential removal of the "init"
> > method. I think it is useful, as the service may have to do some
> > initialization before it goes online. I do not think this method is
> hurting
> > anyone.
> >
> > This thread is getting too long, and I am sure that most readers are
> > already getting lost in the proposed design. I would start a new thread
> > with a summary of all proposed changes.
> >
> > D.
> >
> > On Fri, Apr 20, 2018 at 5:25 PM, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Denis,
> > >
> > > > On the other hand, if exception is thrown from the *execute()
> *method,
> > > then service won't be undeployed.
> > >
> > > This is actually weird... What is going to happen in this case and how
> > user
> > > would handle this?
> > >
> > > -Val
> > >
> > > On Fri, Apr 20, 2018 at 1:10 AM, Denis Mekhanikov <
> dmekhani...@gmail.com
> > >
> > > wrote:
> > >
> > > > Val,
> > > >
> > > > *init()* method is executed before a service is considered deployed.
> > > > If any exception is thrown from it, then it will be handled as
> > deployment
> > > > failure.
> > > >
> > > > *execute() *method is run after the service is deployed, and it can
> > keep
> > > > running until the service is cancelled.
> > > > This method has its own thread, so it can perform some background
> work.
> > > >
> > > > Suppose you want to deploy HTTP server as a service on one of your
> > nodes.
> > > > You can place HTTP server creation logic in the *init() *method.
> > > > If some nodes don't have a permission to listen to needed ports,
> then a
> > > > corresponding exception will be propagated to the user code.
> > > > On the other hand, if exception is thrown from the *execute()
> *method,
> > > then
> > > > service won't be undeployed.
> > > >
> > > > Denis
> > > >
> > > > пт, 20 апр. 2018 г. в 2:35, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Denis,
> > > > >
> > > > > I totally agree with you. I'm just not sure why do we need two
> > methods
> > > > > (init and execute) that have virtually same semantics. With the new
> > > > design,
> > > > > what would be the difference between these methods from user point
> of
> > > > view,
> > > > > and how one would determine what exactly should go in each of them?
> > > > Unless
> > > > > I'm missing something, it looks like unnecessary complication.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Apr 17, 2018 at 1:00 AM, Denis Mekhanikov <
> > > dmekhani...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > Service initialisation is not going to happen in the discovery
> > > thread.
> > > > > > It should be done asynchronously, and initialisation results
> should
> > > be
> > > > > sent
> > > > > > to the coordinator over communication.
> > > > > > This is described in the IEP:
> > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> > > > > > 17%

Re: Topology-wide notification on critical errors

[Dmitriy Setrakyan]

On Fri, Apr 20, 2018 at 4:50 AM, Anton Vinogradov  wrote:

> P.s. Andrey Kuznetsov, corrected me that we have no warranty that failed
> node able to notify cluster.
>
> But,
>
> try{
>sendDiscoveryMessageWithFail(...);
> } catch(){
>// No-op;
> }
>
> is better than nothing, I think.
>

Agree about the "better than nothing" part, but do not agree about the
"no-op" in the catch block. We should still log the fact that sending of
the failure message failed and provide the exception stack trace if there
is one.

Re: Service grid redesign

[Dmitriy Setrakyan]

 along with
> the
> > > > > service
> > > > > > > > > > >> > configurations on all nodes, that would show how
> many
> > > > times
> > > > > > the
> > > > > > > > > > service
> > > > > > > > > > >> > state has changed, i.e. it has been
> > > undeployed/redeployed.
> > > > > It
> > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > >> > kept for undeployed services as well to avoid
> > situations
> > > > > like
> > > > > > I
> > > > > > > > > > >> described.
> > > > > > > > > > >> >
> > > > > > > > > > >> > But it still leaves a possibility of incorrect
> > > behaviour,
> > > > if
> > > > > > > there
> > > > > > > > > > was a
> > > > > > > > > > >> > split-brain situation at some point. I don't think
> we
> > > > should
> > > > > > > > precess
> > > > > > > > > > it
> > > > > > > > > > >> > somehow, though. If we choose to tackle it, it will
> > > > > > > overcomplicate
> > > > > > > > > > >> things
> > > > > > > > > > >> > for a sake of a minor improvement.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Denis
> > > > > > > > > > >> >
> > > > > > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko <
> > > > > > > > > > >> > valentin.kuliche...@gmail.com>:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > I was responding to another Denis :) Agree with
> you
> > on
> > > > > your
> > > > > > > > point
> > > > > > > > > > >> though.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > -Val
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda <
> > > > > > > dma...@apache.org>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Val,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Guess we're talking about other situations. I'm
> > > > bringing
> > > > > > up
> > > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > >> > > when a
> > > > > > > > > > >> > > > service was deployed dynamically and has to be
> > > brought
> > > > > up
> > > > > > > > after
> > > > > > > > > a
> > > > > > > > > > >> full
> > > > > > > > > > >> > > > cluster restart w/o user intervention. To
> achieve
> > > this
> > > > > we
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > >> > persist
> > > > > > > > > > >> > > > the service's configuration somewhere.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > --
> > > > > > > > > > >> > > > Denis
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin
> > Kulichenko
> > > <
> > > > > > > > > > >> > > > valentin.kuliche...@gmail.com> wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Denis,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time
> a
> > > > class
> > > > > is
> > > > > > > > > > deployed
> > > > > > > > > > >> or
> > > > > > > > > > >> > > > > redeployed. If this doesn't happen in some
> > cases,
> > > I
> > > > > > > believe
> > > > > > > > > this
> > > > > > > > > > >> > would
> > > > > > > > > > >> > > > be a
> > > > > > > > > > >> > > > > bug. I don't think we need to add any new
> > events.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > -Val
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda <
> > > > > > > > > dma...@apache.org
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Denis,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > I would encourage us to persist a service
> > > > > > configuration
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >> meta
> > > > > > > > > > >> > > > store
> > > > > > > > > > >> > > > > > and have this capability enabled by default.
> > > > That's
> > > > > > > > > essential
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > > services
> > > > > > > > > > >> > > > > > started dynamically. Moreover, we support
> > > similar
> > > > > > > behavior
> > > > > > > > > for
> > > > > > > > > > >> > > caches,
> > > > > > > > > > >> > > > > > indexes, and other DDL changes happened at
> > > > runtime.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > --
> > > > > > > > > > >> > > > > > Denis
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis
> > > Mekhanikov <
> > > > > > > > > > >> > > > dmekhani...@gmail.com>
> > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Another question, that I would like to
> > discuss
> > > > is
> > > > > > > > whether
> > > > > > > > > > >> > services
> > > > > > > > > > >> > > > > should
> > > > > > > > > > >> > > > > > > be preserved on cluster restarts.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Currently it depends on persistence
> > > > configuration.
> > > > > > If
> > > > > > > > > > >> persistence
> > > > > > > > > > >> > > for
> > > > > > > > > > >> > > > > any
> > > > > > > > > > >> > > > > > > data region is enabled, then services will
> > be
> > > > > > > persisted
> > > > > > > > as
> > > > > > > > > > >> well.
> > > > > > > > > > >> > > This
> > > > > > > > > > >> > > > > is
> > > > > > > > > > >> > > > > > a
> > > > > > > > > > >> > > > > > > pretty strange way of configuring this
> > > > behaviour.
> > > > > > > > > > >> > > > > > > I'm not sure, if anybody relies on this
> > > > > > functionality
> > > > > > > > > right
> > > > > > > > > > >> now.
> > > > > > > > > > >> > > > Should
> > > > > > > > > > >> > > > > > we
> > > > > > > > > > >> > > > > > > support it at all? If yes, should we make
> it
> > > > > > > > configurable?
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Denis
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis
> > Mekhanikov <
> > > > > > > > > > >> > > dmekhani...@gmail.com
> > > > > > > > > > >> > > > >:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > Val,
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that
> user
> > > > > should
> > > > > > > have
> > > > > > > > > > some
> > > > > > > > > > >> way
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > know,
> > > > > > > > > > >> > > > > > > > that new version of a service class was
> > > > > deployed.
> > > > > > > > > > >> > > > > > > > One way to do it is to listen to
> > > > > > > *EVT_CLASS_DEPLOYED.
> > > > > > > > > *I'm
> > > > > > > > > > >> not
> > > > > > > > > > >> > > > sure,
> > > > > > > > > > >> > > > > > > > whether it is triggered on class
> > > redeployment,
> > > > > > > though.
> > > > > > > > > If
> > > > > > > > > > >> not,
> > > > > > > > > > >> > > then
> > > > > > > > > > >> > > > > > > another
> > > > > > > > > > >> > > > > > > > event type should be added.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > I don't think, that a lot of people will
> > > > > implement
> > > > > > > > their
> > > > > > > > > > own
> > > > > > > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make
> work
> > > with
> > > > > > > > > > >> > *UriDeploymentSpi*
> > > > > > > > > > >> > > > as
> > > > > > > > > > >> > > > > > > > comfortable as possible.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Denis
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin
> > > > Kulichenko
> > > > > <
> > > > > > > > > > >> > > > > > > > valentin.kuliche...@gmail.com>:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >> Yes, the class deployment itself has to
> > be
> > > > > > > explicit.
> > > > > > > > > > I.e.,
> > > > > > > > > > >> > there
> > > > > > > > > > >> > > > has
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > be
> > > > > > > > > > >> > > > > > > >> a manual step where user updates the
> > class,
> > > > and
> > > > > > the
> > > > > > > > > exact
> > > > > > > > > > >> step
> > > > > > > > > > >> > > > > > required
> > > > > > > > > > >> > > > > > > >> would depend on DeploymentSpi
> > > implementation.
> > > > > But
> > > > > > > > then
> > > > > > > > > > >> Ignite
> > > > > > > > > > >> > > > takes
> > > > > > > > > > >> > > > > > care
> > > > > > > > > > >> > > > > > > >> of
> > > > > > > > > > >> > > > > > > >> everything else - service redeployment
> > and
> > > > > > restart
> > > > > > > is
> > > > > > > > > > >> > automatic.
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be
> > > > > disabled
> > > > > > if
> > > > > > > > > > >> > > DeploymentSpi
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > > > not
> > > > > > > > > > >> > > > > > > >> configured. In this case service class
> > > > > > definitions
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >> > > > > deployed
> > > > > > > > > > >> > > > > > > on
> > > > > > > > > > >> > > > > > > >> local classpath and can't be updated in
> > > > > runtime.
> > > > > > > Just
> > > > > > > > > > like
> > > > > > > > > > >> it
> > > > > > > > > > >> > > > works
> > > > > > > > > > >> > > > > > > right
> > > > > > > > > > >> > > > > > > >> now.
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> -Val
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM,
> Dmitriy
> > > > > > Setrakyan
> > > > > > > <
> > > > > > > > > > >> > > > > > > dsetrak...@apache.org
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> wrote:
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM,
> Dmitry
> > > > > Pavlov <
> > > > > > > > > > >> > > > > > dpavlov@gmail.com>
> > > > > > > > > > >> > > > > > > >> > wrote:
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > > Hi Igniters,
> > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > >> > > > > > > >> > > I like automatic redeploy which can
> > be
> > > > > > disabled
> > > > > > > > by
> > > > > > > > > > >> config
> > > > > > > > > > >> > if
> > > > > > > > > > >> > > > > user
> > > > > > > > > > >> > > > > > > >> wants
> > > > > > > > > > >> > > > > > > >> > to
> > > > > > > > > > >> > > > > > > >> > > control this process. What do you
> > > think?
> > > > > > > > > > >> > > > > > > >> > >
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > I do not think we should have
> anything
> > > > > > automatic
> > > > > > > > when
> > > > > > > > > > it
> > > > > > > > > > >> > comes
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > > > >> > deployment, everything should be
> > > explicit.
> > > > > > > However,
> > > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > >> > deployment SPI, then a user should be
> > > able
> > > > to
> > > > > > do
> > > > > > > > > "hot"
> > > > > > > > > > >> > > redeploy,
> > > > > > > > > > >> > > > > > > where a
> > > > > > > > > > >> > > > > > > >> > new service will be deployed if the
> > user
> > > > > drops
> > > > > > an
> > > > > > > > > > updated
> > > > > > > > > > >> > jar.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > We should not create anything new
> here.
> > > > > Ignite
> > > > > > > > > already
> > > > > > > > > > >> has a
> > > > > > > > > > >> > > > > > > deployment
> > > > > > > > > > >> > > > > > > >> SPI
> > > > > > > > > > >> > > > > > > >> > and it already works in a certain
> way.
> > > > Let's
> > > > > > not
> > > > > > > > > change
> > > > > > > > > > >> it.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >> > D.
> > > > > > > > > > >> > > > > > > >> >
> > > > > > > > > > >> > > > > > > >>
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Ignite RPM packaging (Stage II)

[Dmitriy Setrakyan]

;>>>>>
> > >>>>>>>>>>> Hello!
> > >>>>>>>>>>>
> > >>>>>>>>>>> Let me share my idea of how this shoud work. Splitting
> package
> > >>>>>> into
> > >>>>>>>>>>> sub-packages should be dependency-driven.
> > >>>>>>>>>>>
> > >>>>>>>>>>> It means that all Ignite modules without dependencies or with
> > >>>>>> small
> > >>>>>>>>>>> dependencies (such as ignite-log4j) should be included in
> > >>>>>>> ignite-core.
> > >>>>>>>>> It
> > >>>>>>>>>>> doesn't make sense to make a zillion RPM packages.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Critical things like ignite-spring and ignite-indexing should
> > >>>> be
> > >>>>>> in
> > >>>>>>>>>>> ignite-core of course, even if they have dependencies.
> > >>>>> Ignite-core
> > >>>>>>>>> should
> > >>>>>>>>>>> be fully self-sufficient and feature-complete.
> > >>>>>>>>>>>
> > >>>>>>>>>>> However, e.g. .net API should probably be in a separate
> > >>>> package,
> > >>>>>>>>> because it
> > >>>>>>>>>>> should depend on mono | net-core. We may also have
> > >>>> ignite-devel
> > >>>>>>>> package
> > >>>>>>>>>>> which should include all modules which only make sense for
> > >>>>>>> developers
> > >>>>>>>>> who
> > >>>>>>>>>>> write code. Such as hibernate integration.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'm not sure about MR modules. The main question should be,
> > >>>> does
> > >>>>>> it
> > >>>>>>>> have
> > >>>>>>>>>>> dependencies? Can it run stand-alone without writing code?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hope this helps,
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Ilya Kasnacheev
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2018-03-27 15:10 GMT+03:00 Petr Ivanov  >:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Igniters!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Here are some news on our RPM packages initiative.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. I’ve finished preliminary developing of Stage II version
> > >>>> of
> > >>>>>> RPM
> > >>>>>>>>>>>> packages [1]. Main “new feature” is — split design. Also
> I’ve
> > >>>>>> added
> > >>>>>>>>>>>> package.sh script for automating package building process
> > >>>> which
> > >>>>>>> will
> > >>>>>>>>> help
> > >>>>>>>>>>>> organise corresponding builds in TC as well as simplify
> > >>>> process
> > >>>>>> for
> > >>>>>>>>>>>> developers who wishes to have custom packages.
> > >>>>>>>>>>>> PR#3703 [2] is ready for review. Denis, in order to catch up
> > >>>>> with
> > >>>>>>>>> Apache
> > >>>>>>>>>>>> Ignite 2.5 release, I’d greatly appreciate your help in
> > >>>> finding
> > >>>>>>>>> reviewer.
> > >>>>>>>>>>>> 2. With the help of ASF INFRA team, we now have RPM [3] and
> > >>>> DEB
> > >>>>>> [4

Re: Topology-wide notification on critical errors

[Dmitriy Setrakyan]

On Thu, Apr 19, 2018 at 8:19 AM, Yakov Zhdanov  wrote:

> Guys,
>
> We have activity to implement a set of mechanisms to handle critical issues
> on nodes (IEP-14 - [1]).
>
> I have an idea to spread message about critical issues to nodes through
> entire topology and put it to logs of all nodes. In my view this will add
> much more clarity. Imagine all nodes output message to log - "Critical
> system thread failed on node XXX [details=...]". This should help a lot
> with investigations.
>
> Andrey Gura, Alex Goncharuk what do you think?
>

Yakov, even though you did not ask me what I think, but I really like the
idea :)

Re: Triggering rebalancing on timeout or manually if the baseline topology is not reassembled

[Dmitriy Setrakyan]

On Tue, Apr 17, 2018 at 4:38 PM, Denis Magda  wrote:

> Dmitriy,
>
> We don't want to disable the baseline topology for the scenario discussed
> here. The goal is to make it more flexible by triggering the rebalancing in
> some circumstances.
>
> As for the SAFE or AGGRESSIVE policies, haven't seen the discussion on the
> dev. So, not sure what it's intended for (use case, scenario, behavior).
>

The idea was to have a mode for disabling BLT, so users will not have to
provide coding callbacks to force the rebalance. Essentially, if we had
this policy, you would not need to provide this example.

Re: Triggering rebalancing on timeout or manually if the baseline topology is not reassembled

[Dmitriy Setrakyan]

On Tue, Apr 17, 2018 at 10:47 AM, Denis Magda  wrote:

> Thanks, Pavel!
>
> Alexey, Ivan, could you check that there are no any pitfalls in the example
> and it can be used as a template for our users?
> https://issues.apache.org/jira/secure/attachment/
> 12919452/BaselineWatcher.java


Denis, I think the proper fix is to add ability to the product to disable
BLT (baseline topology). I remember seeing a discussion in have SAFE and
AGGRESSIVE (of SAFE and NONE) policies for BLT. Do we have a ticket for it?

D.

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

On Thu, Apr 12, 2018 at 9:45 AM, Ivan Rakov  wrote:

> Dmitriy,
>
> fsync() is really slow operation - it's the main reason why FSYNC mode is
> way slower than LOG_ONLY.
> Fix includes extra fsyncs in necessary parts of code and nothing more.
> Every part is important - at the beginning of the thread I described why.
>
> 20% slow in benchmark doesn't mean than Ignite itself will become 20%
> slower. Benchmark replays only "data loading" scenario. It signals that
> maximum throughput with WAL enabled will be 20% slower. By the way, we
> already have option to disable WAL in runtime for the period of data
> loading.
>
>
Ivan, I get it, but I am sure that you can do more things in parallel. Do
we wait for the fsync call to complete? If yes, do we have to wait? Are
there other performance optimizations you can add, considering that we are
in LOG_ONLY or BACKGROUND modes and disk writes may be delayed.

D.

Re: Apache Ignite RPM packaging (Stage II)

[Dmitriy Setrakyan]

t;
> > >>> Also — a question arose while I was working on this issue: which OSes
> > (and
> > >>> which versions of each) are we going to support (if we are going) in
> > terms
> > >>> of step-by-step list? Currently RPM packages are tested only with
> > latest
> > >>> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> > >>> distributives [6] some of which are more o less popular among OS
> > community
> > >>> (ALT, Fedora, openSUSE, etc.).
> > >>>
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> > >>> [2] https://github.com/apache/ignite/pull/3703
> > >>> [3] https://bintray.com/apache/ignite-rpm
> > >>> [4] https://bintray.com/apache/ignite-deb
> > >>> [5] https://bintray.com/apache/cassandra/debian#files/
> > >>> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_
> > distributions
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>> On 15 Mar 2018, at 22:15, Petr Ivanov  wrote:
> > >>>>
> > >>>> I suppose that most everything if not all from libs/options will go
> to
> > >>> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > >>>> More precise lib selection (if something from optional would better
> to
> > >>> have in core package) will be discussed right after preliminary split
> > >>> architecture agreement.
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On 15 Mar 2018, at 22:11, Dmitry Pavlov 
> > wrote:
> > >>>>>
> > >>>>> I like idea of keeping simple system of modules, so +1 from me.
> > >>>>>
> > >>>>> Where optional libs (e.g Direct IO plugin) would be included, would
> > it
> > >>> be
> > >>>>> core or optional?
> > >>>>>
> > >>>>> чт, 15 мар. 2018 г. в 22:09, Denis Magda :
> > >>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> How big would be a final core module?
> > >>>>>>> Around 30M. Can be shrinked to ~15M if separate Visor and create
> > it’s
> > >>> own
> > >>>>>>> package.
> > >>>>>>
> > >>>>>>
> > >>>>>> Guys, 30 vs 280M is a hge difference.  I would agree with Petr
> > and
> > >>>>>> propose the simplest modular system:
> > >>>>>>
> > >>>>>> - core module that includes basic Ignite capabilities including
> SQL,
> > >>>>>> compute grid, service grid, k/v
> > >>>>>> - optional module hosts the rest - ML, streamers integration
> (kafka,
> > >>>>>> flink), kubernetes, etc.
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Denis
> > >>>>>>
> > >>>>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov <
> mr.wei...@gmail.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> *DEB package
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 15 Mar 2018, at 10:35, Petr Ivanov 
> > wrote:
> > >>>>>>>>
> > >>>>>>>> Considering that DEV package for now is almost platform
> > independent
> > >>>>>> (its
> > >>>>>>> a java application more or less), that package will work almost
> on
> > any
> > >>>>>>> DEB-based linux, including but not limited to Ubuntu, Debian,
> etc.
> > >>>>>>>> The only restriction is existence of systemctl (systemd) service
> > >>>>>> manager
> > >>>>>>> — we are dependent on it.
> > >>>>>>>>
> > >>>>>>>> Thats why, for instance, our RPM repository is called simply
> ‘rpm’
> > >>> and
> > >>>>>>> package has no arch or dist suffix — it will work on CentOS,
> RHEL,
> > >>>>&g

Re: Ignite documentation is broken

[Dmitriy Setrakyan]

This is what I see (attached)

On Wed, Apr 11, 2018 at 1:52 AM, Alexey Kuznetsov 
wrote:

> I have no problems.
>
> Try other browser  / or force refresh of page.
>
> On Wed, Apr 11, 2018 at 3:49 PM, Dmitriy Setrakyan 
> wrote:
>
> > Igniters,
> >
> > The readme documentation seems broken. Is it only for me, or others
> > experience the same thing?
> >
> > https://apacheignite.readme.io/docs
> >
> > Did anyone change anything in the docs settings?
> >
> > D.
> >
>
>
>
> --
> Alexey Kuznetsov
>

Ignite documentation is broken

[Dmitriy Setrakyan]

Igniters,

The readme documentation seems broken. Is it only for me, or others
experience the same thing?

https://apacheignite.readme.io/docs

Did anyone change anything in the docs settings?

D.

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 11:57 PM, Ilya Suntsov 
wrote:

> Dmitriy,
>
> I've measured performance on the current master and haven't found any
> problems with in-memory mode.
>

Got it. I would still say that the performance drop is too big with
persistence turned on. It seems like we did not just fix the bug, we also
introduced some additional slow down there. I would investigate if we could
optimize.

Re: Warning message

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 2:22 PM, Alew  wrote:

> Hi!
>
> I use local cache and get a lot of warnings when execute ScanQueries.
>
> [13:26:25 WRN]  Ignoring query projection because it's
> executed over LOCAL cache (only local node will be queried):
> GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null,
> filter=o.a.i.i.processors.platform.cache.PlatformCacheEntryF
> ilterImpl@7629939a, transform=null, part=null, incMeta=false,
> metrics=GridCacheQueryMetricsAdapter [minTime=9223372036854775807,
> maxTime=0, sumTime=0, avgTime=0.0, execs=0, completed=0, fails=0],
> pageSize=1024, timeout=0, keepAll=true, incBackups=false, dedup=false,
> prj=o.a.i.i.cluster.ClusterGroupAdapter@708472f7, keepBinary=true,
> subjId=null, taskHash=0]
>
> What does it warn me about? Is it really warning?
>

Your cache is LOCAL, i.e. not distributed. This means that all the data is
located only on the local node. More on cache modes here:
https://apacheignite.readme.io/docs/cache-modes

D.

Re: Atomic caches

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 2:03 PM, akurbanov  wrote:

> Dmitry,
>
> Sorry for confusing topic. I I'm pretty sure that configuration for atomic
> caches is validated, will double-check this. I was referring only atomic
> data structures cache.
>

Got it. We should definitely add validation for the atomic data structures
configuration as well.

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

I am not convinced that the performance degradation is only due to the new
change that fixes the incorrect behavior. To my knowledge, there is also a
drop in memory-only mode. Can someone explain why do we have such a drop?

D.

On Tue, Apr 10, 2018 at 9:08 AM, Vladimir Ozerov 
wrote:

> 16% looks perfectly ok to me provided that we compare correct
> implementation with incorrect one.
>
> вт, 10 апр. 2018 г. в 18:24, Dmitriy Setrakyan :
>
> > Ilya, can we find out why pure in-memory scenario also had a performance
> > drop and which commit caused it? It should not be affected by changes in
> > persistence at all.
> >
> > D.
> >
> > On Tue, Apr 10, 2018 at 7:56 AM, Ilya Suntsov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Looks like commit:
> > >
> > > d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > > > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > > > Author: Ivan Rakov 
> > > > Date:   Tue Mar 27 20:11:52 2018 +0300
> > > > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on
> > checkpoint
> > > > begin - Fixes #3656.
> > >
> > >
> > > was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> > > benchmarks (LOG_ONLY):
> > >
> > >- atomic-put  (16 %)
> > >- atomic-putAll (14 %)
> > >- tx-putAll (11 %)
> > >
> > > As I understand it is greater than initial assessment.
> > >
> > > Thoughts?
> > >
> > > 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
> > >
> > > > Ivan, sure :)
> > > >
> > > > Thank you for this contribution, merged to master.
> > > >
> > > > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> > > >
> > > > > Dmitry,
> > > > >
> > > > > Firstly PR contained dirty fix for performance measurement, but now
> > it
> > > > > contains good fix. :) Sorry for inconvenience.
> > > > > I've renamed the PR.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > > > Hi Eduard, thank you for review.
> > > > > >
> > > > > > Hi Ivan,
> > > > > >
> > > > > > I'm confused on PR naming
> > > > > > https://github.com/apache/ignite/pull/3656
> > > > > >
> > > > > > Could you rename?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > > > eduard.shangar...@gmail.com
> > > > > >> :
> > > > > >> Ivan, I have reviewed your changes, looks good.
> > > > > >>
> > > > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov <
> > ivan.glu...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >>> Igniters,
> > > > > >>>
> > > > > >>> I've completed development of https://issues.apache.org/jira
> > > > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> > > changes.
> > > > > >>> Please note that it will be possible to track time of WAL fsync
> > on
> > > > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> > > "Checkpoint
> > > > > >>> started" message.
> > > > > >>>
> > > > > >>> Also, I've created https://issues.apache.org/
> > > jira/browse/IGNITE-8057
> > > > > >> with
> > > > > >>> description of possible further improvement of WAL fsync on
> > > > checkpoint
> > > > > >>> begin.
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>> Ivan Rakov
> > > > > >>>
> > > > > >>>
> > > > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > > > >>>
> > > > > >>>> Ivan,
> > > > > >>>>
> > > > > >>>> It's all good then :) Thanks!
> > > > > &g

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

;> provided
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> performance results.
> > > >>>>>>>>>>>>>>>> 2018-03-21 18:20 GMT+03:00 Vladimir Ozerov <
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> voze...@gridgain.com
> > > >>>>>>>>>>>>>> :
> > > >>>>>>>>>> +1 for accepting drop in LOG_ONLY. 7% is not that much and
> > > >>>>>>>>>>>> not a
> > > >>>>>>>>>>>>>> drop
> > > >>>>>>>>> at
> > > >>>>>>>>>>>>>>>> all, provided that we fixing a bug. I.e. should we
> > > implement
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>> correctly
> > > >>>>>>>>> in the first place we would never notice any "drop".
> > > >>>>>>>>>>>>>>>> I do not understand why someone would like to use
> > current
> > > >>>>>>>>>>>>>>>>> broken
> > > >>>>>>>>>>>>>>> mode.
> > > >>>>>>>>>> On Wed, Mar 21, 2018 at 6:11 PM, Dmitry Pavlov
> > > >>>>>>>>>>>>>>>>> 
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi, I think option 1 is better. As Val said any mode
> > that
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> allows
> > > >>>>>>>>>>>>>>> corruption
> > > >>>>>>>>>> does not make much sense.
> > > >>>>>>>>>>>>>>>>>> What Ivan mentioned here as drop, in relation to old
> > > mode
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> DEFAULT
> > > >>>>>>>>>>>>>>>> (FSYNC
> > > >>>>>>>>>>> now), is still significant perfromance boost.
> > > >>>>>>>>>>>>>>>>> Sincerely,
> > > >>>>>>>>>>>>>>>>>> Dmitriy Pavlov
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> ср, 21 мар. 2018 г. в 17:56, Ivan Rakov <
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> ivan.glu...@gmail.com
> > > >>>>>>>>>>>>>>>> :
> > > >>>>>>>>>> I've attached benchmark results to the JIRA ticket.
> > > >>>>>>>>>>>> We observe ~7% drop in "fair" LOG_ONLY_SAFE mode,
> > > >>>>>>>>>>>>>>>>>>> independent
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>> WAL
> > > >>>>>>>>>>> compaction enabled flag. It's pretty significant drop: WAL
> > > >>>>>>>>>>>>>>>>> compaction
> > > >>>>>>>>>>>>>>>>> itself gives only ~3% drop.
> > > >>>>>>>>>>>>>>>>> I see two options here:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 1) Change LOG_ONLY behavior. That implies that we'll
> > be
> > > >>>>>>>>>>>>>>>>>>> ready
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>> release
> > > >>>>>>>>>>> AI 2.5 with 7% drop.
> > > >>>>>>>>>>>>>>>>>> 2) Introduce LOG_ONLY_SAFE, make it default, add
> > release
> > > >>>>>>>>>>>>>>>>>>> note
> > > >>>>>>>>>>>>>>>>> to AI
> > > >>>>>>>>> 2.5
> > > >>>>>>>>>>>>>>>>>> that we added power loss durability in default mode,
> > but
> > > >>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>> fallback to previous LOG_ONLY in order to retain
> > > >>>>>>>>>> performance.
> > > >>>>>>>>>>>>>>>>> Thoughts?
> > > >>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>> Ivan Rakov
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 20.03.2018 16:00, Ivan Rakov wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Val,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> If a storage is in
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> corrupted state, does it mean that it needs to be
> > > >>>>>>>>>>>>>>>>>>>>> completely
> > > >>>>>>>>>>>>>>>>>>> removed
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>> cluster needs to be restarted without data?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Yes, there's a chance that in LOG_ONLY all local
> data
> > > >> will
> > > >>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>> lost,
> > > >>>>>>>>>> but only in *power loss**/ OS crash* case.
> > > >>>>>>>>>>>>>>>>> kill -9, JVM crash, death of critical system thread
> and
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>>> other
> > > >>>>>>>>> cases that usually take place are variations of *process
> > > >>>>>>>>>>>>>>>>>>>> crash*.
> > > >>>>>>>>>>>>>>>>>> All
> > > >>>>>>>>>>> WAL modes (except NONE, of course) ensure corruption-safety
> > > >>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>> case
> > > >>>>>>>>> of
> > > >>>>>>>>>>>>>>>>> process crash.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> If so, I'm not sure any mode
> > > >>>>>>>>>>>>>>>>>>>> that allows corruption makes much sense to me.
> > > >>>>>>>>>>>>>>>>>>>>> It depends on performance impact of enforcing
> > > >> power-loss
> > > >>>>>>>>>>>>>>>>>>>> corruption
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> safety. Price of full protection from power loss is
> > > high
> > > >> -
> > > >>>>>>>>>>>>>>>>> FSYNC
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>> way slower (2-10 times) than other WAL modes. The question is
> > > >>>>>>>>>>>>>>>>> whether
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> ensuring weaker guarantees (corruption can't happen,
> > but
> > > >>>>>>>>>>>>>>>>>> loss
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>> last
> > > >>>>>>>>>> updates can) will affect performance as badly as strong
> > > >>>>>>>>>>>>>>>>>> guarantees.
> > > >>>>>>>>>>>>>>>>>> I'll share benchmark results soon.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Ivan Rakov
> > > >>>>>>>>>>>>>>>>>>>> On 20.03.2018 5:09, Valentin Kulichenko wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Guys,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> What do we understand under "data corruption"
> here?
> > > If
> > > >> a
> > > >>>>>>>>>>>>>>>>>>>>> storage
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>> corrupted state, does it mean that it needs to be
> > > >> completely
> > > >>>>>>>>>>>>>>>>>> removed
> > > >>>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> cluster needs to be restarted without data? If so,
> I'm
> > > not
> > > >>>>>>>>>>>>>>>>>> sure
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> any
> > > >>>>>>>>>> mode
> > > >>>>>>>>>>>>>>>>>> that allows corruption makes much sense to me. How
> am
> > I
> > > >>>>>>>>>>>>>>>>>> supposed
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>> use a
> > > >>>>>>>>>>>>>>>>>> database, if virtually any failure can end with
> > complete
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> loss of
> > > >>>>>>>>>>>>>>>>>>>>> data?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> In any case, this definitely should not be a
> default
> > > >>>>>>>>>>>>>>>>>> behavior.
> > > >>>>>>>>>>>>>>>> If
> > > >>>>>>>>> user ever
> > > >>>>>>>>>>>>>>>>>> switches to corruption-unsafe mode, there should be
> a
> > > >>>>>>>>>>>>>>>>>> clear
> > > >>>>>>>>>>>>>>>>>>> warning
> > > >>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>> this.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> -Val
> > > >>>>>>>>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 1:06 AM, Ivan Rakov <
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> ivan.glu...@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>> Ticket to track changes:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-7754
> > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>> Ivan Rakov
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On 16.03.2018 10:58, Dmitriy Setrakyan wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Fri, Mar 16, 2018 at 12:55 AM, Ivan Rakov <
> > > >>>>>>>>>>>>>>>>>>>>>> ivan.glu...@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>> Vladimir,
> > > >>>>>>>>>>>>>>>>>>>> Unlike BACKGROUND, LOG_ONLY provides strict write
> > > >>>>>>>>>>>>>>>>>>>>>>> guarantees
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> unless power
> > > >>>>>>>>>>> loss has happened.
> > > >>>>>>>>>>>>>>>>>>>>>>>> Seems like we need to measure performance
> > > difference
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> decide
> > > >>>>>>>>> whether do
> > > >>>>>>>>>>>>>>>>>>>>> we need separate WAL mode. If it will be
> invisible,
> > > >>>>>>>>>>>>>>>>>> we'll
> > > >>>>>>>>>>>>>>>>>>>>>> just
> > > >>>>>>>>>> fix
> > > >>>>>>>>>>>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>>>>> bugs without introducing new mode; if it will be
> > > >>>>>>>>>>>>>>>>>>>> perceptible,
> > > >>>>>>>>>>>>>>>>>>>>>>>> we'll
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> continue the discussion about introducing
> > > >>>>>>>>>>>>>>>>>>>>>> LOG_ONLY_SAFE.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Makes sense?
> > > >>>>>>>>>>>>>>>>>>>> Yes, this sounds like the right approach.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > >
> > >
> >
>
>
>
> --
> Best Regards,
> Ilya Suntsov
> email: isunt...@gridgain.com
> *GridGain Systems*
> www.gridgain.com
>

Re: Remove cache groups in AI 3.0

[Dmitriy Setrakyan]

Vladimir, sounds like a huge refactoring. Other than "cache groups are
confusing", are we solving any other big issues with the new proposed
approach?

(every time we try to refactor rebalancing, I get goose bumps)

D.

On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> Cache groups were implemented for a sole purpose - to hide internal
> inefficiencies. Namely (add more if I missed something):
> 1) Excessive heap usage for affinity/partition data
> 2) Too much data files as we employ file-per-partition approach.
>
> These problems were resolved, but now cache groups are a great source of
> confusion both for users and us - hard to understand, no way to configure
> it in deterministic way. Should we resolve mentioned performance issues we
> would never had cache groups. I propose to think we would it take for us to
> get rid of cache groups.
>
> Please provide your inputs to suggestions below.
>
> 1) "Merge" partition data from different caches
> Consider that we start a new cache with the same affinity configuration
> (cache mode, partition number, affinity function) as some of already
> existing caches, Is it possible to re-use partition distribution and
> history of existing cache for a new cache? Think of it as a kind of
> automatic cache grouping which is transparent to the user. This would
> remove heap pressure. Also it could resolve our long-standing issue with
> FairAffinityFunction when tow caches with the same affinity configuration
> are not co-located when started on different topology versions.
>
> 2) Employ segment-extent based approach instead of file-per-partition
> - Every object (cache, index) reside in dedicated segment
> - Segment consists of extents (minimal allocation units)
> - Extents are allocated and deallocated as needed
> - *Ignite specific*: particular extent can be used by only one partition
> - Segments may be located in any number of data files we find convenient
> With this approach "too many fsyncs" problem goes away automatically. At
> the same time it would be possible to implement efficient rebalance still
> as partition data will be split across moderate number of extents, not
> chaotically.
>
> Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> they?
>
> Vladimir.
>

Re: REST API and new authentication API

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 12:28 AM, Alexey Kuznetsov 
wrote:

> Dmitriy,
>
> Yes, because we have a command "Add new user" and this command can be
> executed only with credentials of some "admin" user.
>
> It means, that in one command you need to specify name of new user and
> "admin" credentials at the same time.?


> If you have any ideas how we can handle this - I will be glad to discuss
> it.
>

I am not sure if I agree with the approach you have suggested. In my view,
we should have "authenticate" command, which should ask for the username
and password. Once the user is authenticated and logged in, you should use
the session token to perform all other commands. We should NOT be
authenticating users on every command.

If you follow this approach, then the command for adding a new user should
require any authentication.

Makes sense?

D.

Re: REST API and new authentication API

[Dmitriy Setrakyan]

Alexey, are you suggesting that we have "newUser" as command parameter, while 
"user" is also a valid command parameter? 

⁣D.​

On Apr 10, 2018, 12:00 AM, at 12:00 AM, Alexey Kuznetsov 
 wrote:
>I looked into code and I think that we could do the following:
>
>1) Use user and password for authentication.
>2) Use newUser and newPassword for new authentication API (add, remove
>and
>update user).
>3) Debug why sessionToken is null.
>
>Created issues:
>https://issues.apache.org/jira/browse/IGNITE-8201
>https://issues.apache.org/jira/browse/IGNITE-8202
>
>I will try to implement them in a couple of days.
>
>
>On Tue, Apr 10, 2018 at 11:32 AM, Alexey Kuznetsov
>
>wrote:
>
>> Igniters,
>>
>> Recently new authentication API was added.
>>
>> I added support for it on REST, but several problems appeared:
>>
>> 1) "&ignite.login=login" and "&ignite.password=pwd" should be used
>for
>> credentials.
>> May be we should use "&user=user" and " &password=pwd " instead?
>> But this will lead to conflict with new authentication API:
>>  For example add user:  http://localhost/ignite?cmd=*adduser*&*user*=
>> sample&*password*=sample&ignite.login=ignite&ignite.password=ignite
>>
>> Any ideas how to resolve this?
>>
>> 2) For some reason when new authentication is enabled session toked
>> returned as null
>>
>{"successStatus":0,*"sessionToken":null*,"error":null,"response":"2.5.0"}
>>
>> Session token can be used instead of adding user name and password to
>> every request.
>>
>> Who can help with resolving this issue?
>>
>> --
>> Alexey Kuznetsov
>>
>
>
>
>--
>Alexey Kuznetsov

Re: Atomic caches

[Dmitriy Setrakyan]

On Mon, Apr 9, 2018 at 6:29 AM, Ilya Lantukh  wrote:

> Anton,
>
> Please do not use term "atomic cache" for system caches that hold internal
> data for atomic data structures. This is very confusing.
>
> You are right, currently there is no logic that will validate cache
> configuration. It definitely should be fixed. And having configuration
> parameters encoded in cache name, like it is currently implemented for
> collections, is one of the most straightforward approaches.
>

Anton, I am completely confused now. Does Ignite validate the configuration
for the atomic caches (not atomic data structures)?

Re: IGNITE-6827 - Review needed.

[Dmitriy Setrakyan]

On Mon, Apr 9, 2018 at 5:42 AM, Alexey Goncharuk  wrote:

> Guys,
>
> After the review in Upsource the configuration parameter was renamed
> to txTimeoutOnPartMapSync, and it makes sense to me because PME is an
> implementation detail and it may change in future, partition map sync is a
> more abstract term. For the same reason I like this parameter being placed
> on transactions configuration - we do not have any parameters for PME, so
> the configuration property goes to an object which affects a user-exposed
> API.
>

AG, are we going to have any other timeouts on PME, like locks? If yes,
then I would still vote of adding PmeTimeout property.

Re: Atomic caches

[Dmitriy Setrakyan]

I would say absolutely YES - we need to have configuration validation.

Igniters, why was the validation skipped in atomic caches?

D.

On Fri, Apr 6, 2018 at 1:43 PM, akurbanov  wrote:

> Hello Igniters,
>
> I want to address a question on AtomicConfiguration validation. I've tested
> in ignite-1.8 branch that it is impossible to start two nodes with
> different
> AtomicConfiguration parameters e.g. different cache modes or numbers of
> backups are provided.
> JIRA link: https://issues.apache.org/jira/browse/IGNITE-2096
>
> In ignite-2.4 AtomicConfiguration validation is completely skipped on node
> startup and this issue is non-reproducible. Node with alternative
> configuration successfully joins, but this configuration is being
> completely
> ignored, all created atomics will reference the same initial configuration
> and belong to the same cache "ignite-sys-atomic-cache@default-ds-group",
> even if configuration is provided in constructor.
>
> Do we need any kind of validation for this configuration?
> Would it be correct to use the same approach for atomic types instances
> caching as used for IgniteQueue/IgniteSet, cache for each unique
> configuration?
>
> Best regards,
> Anton Kurbanov
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Service grid redesign

[Dmitriy Setrakyan]

On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov  wrote:

> Hi Igniters,
>
> I like automatic redeploy which can be disabled by config if user wants to
> control this process. What do you think?
>

I do not think we should have anything automatic when it comes to
deployment, everything should be explicit. However, if we use the
deployment SPI, then a user should be able to do "hot" redeploy, where a
new service will be deployed if the user drops an updated jar.

We should not create anything new here. Ignite already has a deployment SPI
and it already works in a certain way. Let's not change it.

D.

Re: IEP-18: Transparent Data Encryption

[Dmitriy Setrakyan]

Here is a correct link to IEP:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption

On Thu, Apr 5, 2018 at 12:01 PM, Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> Based on previous discussion [1] we've created "IEP-18: Transparent Data
> Encryption" [2]
> I've planned to start implementation of TDE in few weeks.
> I will create JIRA ticket for each piece of implementation.
>
> So, please, see IEP-18 and give us feedback.
>
> Dima Ryabov, huge thanks for pushing TDE IEP forward.
>
> [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/Transparent-Data-Encryption-TDE-in-Apache-Ignite-td18957.html
> [2] https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=75979078
>

Re: Breaking change in JDBC connection string format

[Dmitriy Setrakyan]

Vladimir, my older email got kind of lost. Can you please clarify, will we
be able to support both, older and newer formats, to avoid a breaking
compatibility change between releases?

D.

On Thu, Apr 5, 2018 at 5:53 AM, Vladimir Ozerov 
wrote:

> I think colon is not very good candidate as it clashes with other
> properties (e.g. host-port delimiter). Semicolon looks good to me. I'll
> created a ticket [1] to address this.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8148
>
> On Wed, Apr 4, 2018 at 12:35 AM, Andrey Gura  wrote:
>
> > Denis,
> >
> > actually params (key-value pairs) are separated by colon in provided
> > JIRA issue comment.
> >
> > On Tue, Apr 3, 2018 at 7:55 PM, Denis Magda  wrote:
> > > Vladimir, Taras,
> > >
> > > Will "@" work for us as the delimiter? I would agree with Andrey to
> reuse
> > > this approach for the thin client.
> > >
> > > As for the breaking changes, if update the delimiter for the next
> driver
> > > version and make sure that version understands 2 delimiters for some
> time
> > > (& and the new one), then this would be ideal and not disrupting.
> > >
> > > --
> > > Denis
> > >
> > > On Tue, Apr 3, 2018 at 2:39 AM, Andrey Gura  wrote:
> > >
> > >> Hi,
> > >>
> > >> We've been solve this problem during JDBC2 driver implementation. And
> > >> I don't know any complains about connection string format. Why we can
> > >> just use the same approach? [1]
> > >>
> > >> [1] https://issues.apache.org/jira/browse/IGNITE-1250?
> > >> focusedCommentId=14706511&page=com.atlassian.jira.
> > >> plugin.system.issuetabpanels:comment-tabpanel#comment-14706511
> > >>
> > >> On Tue, Apr 3, 2018 at 12:20 PM, Ilya Kasnacheev
> > >>  wrote:
> > >> > Hello!
> > >> >
> > >> > I think semicolon is the way to go, since round brackets are often
> > >> > interpreted by shells and will need escaping on their own. Let's get
> > rid
> > >> of
> > >> > & and ?.
> > >> >
> > >> > jdbc:ignite:thin://host:port/schema:param1=val1:param2=val2
> > >> >
> > >> > --
> > >> > Ilya Kasnacheev
> > >> >
> > >> > 2018-04-03 11:30 GMT+03:00 Vladimir Ozerov :
> > >> >
> > >> >> Igniters,
> > >> >>
> > >> >> Thanks to Alex Kuznetsov, we revealed serious usability issue in
> our
> > >> thin
> > >> >> JDBC driver - we user ampersand character to delimit properties:
> > >> >>
> > >> >> jdbc:ignite:thin://host:port/schema?param1=val1*&*param2=val2
> > >> >>
> > >> >> This leads to multiple problems:
> > >> >> 1) This format is unusable from many console environments and
> require
> > >> >> special escaping (PowerShell, bash)
> > >> >> 2) Also this causing issues when writing connection string is
> passed
> > to
> > >> >> some 3rd-party applications as sometimes they cannot escape it as
> > well.
> > >> >>
> > >> >> I performed investigation on how other vendors do that. Bottom
> line -
> > >> >> nobody use ampersand. Instead semicolon or parentheses are used:
> > >> >>
> > >> >> jdbc:ignite:thin://host:port/schema?param1=val1¶m2=val2
> > >> >> jdbc:ignite:thin://host:port/schema?(param1=val1)(param2=val2)
> > >> >>
> > >> >> I propose to do a breaking change in AI 2.5 and change the format
> to
> > >> >> *parentheses*. This would be a disruptive experience for existing
> > users,
> > >> >> but in the long term benefits will outweight. Also remember that we
> > do
> > >> not
> > >> >> offered officially any backward compatibility for our JDBC driver.
> > >> >>
> > >> >> Alternatively we may allow users to use previous format with help
> of
> > >> system
> > >> >> property or environment variable.
> > >> >>
> > >> >> Thoughts?
> > >> >>
> > >> >> Vladimir.
> > >> >>
> > >>
> >
>

Re: Memory usage per cache

[Dmitriy Setrakyan]

On Wed, Apr 4, 2018 at 5:27 AM, Alexey Goncharuk  wrote:

> Denis,
>
> I think this particular metric should be deprecated. The most we can do
> about it is to return the actual allocated size when a cache is the only
> cache in a group and return -1 if there are multiple caches in a group.
> However, this does not look like a consistent approach to me, so I would
> prefer to always return -1 and suggest that users use corresponding cache
> group metrics.
>

Why not return cache group metrics from this method by default and properly
document it?

Re: Service grid redesign

[Dmitriy Setrakyan]

Here is the correct link:
https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid

I have looked at the tickets there, and I believe that we should not
support peer-deployment for services. It is very hard and I do not think we
should even try.

I am proposing closing this ticket as Won't Fix -
https://issues.apache.org/jira/browse/IGNITE-975

D.

On Wed, Apr 4, 2018 at 5:39 AM, Denis Mekhanikov 
wrote:

> Vyacheslav,
>
> I've just posted my first draft of the IEP:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Service+grid+
> improvements
> It's not finished yet, but you can get the idea from it.
> If you have some thoughts on your mind, please let me know, I'll add them
> to the IEP.
>
> Denis
>
> ср, 4 апр. 2018 г. в 13:09, Vyacheslav Daradur :
>
> > Denis, thanks for the link.
> >
> > I looked through the task and I think that understand your redesign point
> > now.
> >
> > Do you have a clear plan or IEP for the whole redesign?
> >
> > I'm interested in this component and I'd like to take part in the
> > development.
> >
> >
> >
> > On Mon, Apr 2, 2018 at 2:55 PM, Denis Mekhanikov 
> > wrote:
> > > Vyacheslav,
> > >
> > > Service deployment design, based on replicated utility cache has proven
> > to
> > > be unstable and deadlock-prone.
> > > You can find a list of JIRA issues, connected to it, in my previous
> > letter.
> > >
> > > The intention behind it is similar to the binary metadata redesign,
> that
> > > happened in the following ticket: IGNITE-4157
> > > 
> > > This change in service deployment procedure will eliminate need for
> > another
> > > internal replicated cache
> > > and make service deployment more reliable on unstable topology.
> > >
> > > Denis
> > >
> > > вт, 27 мар. 2018 г. в 23:21, Vyacheslav Daradur :
> > >
> > >> Hi, Denis Mekhanikov!
> > >>
> > >> As far as I know, Ignite services are based on IgniteCache and we have
> > >> all its features. We can use listeners or continuous queries for
> > >> deployment synchronizations.
> > >>
> > >> Why do you want using the discovery layer for that?
> > >>
> > >> One more thing: we can use baseline approach for services, that means
> > >> *IgniteService.deploy()* returns ready to work service after
> > >> deployment on baseline nodes and deploy to other nodes on demand, for
> > >> example when deployed service's loading will be hight.
> > >>
> > >> About versioning, maybe there is sense to extend public API:
> > >> IgniteServices.service(name, *version*)?
> > >>
> > >> At first deployment, we can compute service's hashcode (just for an
> > >> example) and store it, after new deployment request for services with
> > >> an existing name we will compute new service's hashcode and compare
> > >> them if they have different hashcodes that we will deploy new service
> > >> as service with a different version.
> > >>
> > >>
> > >> On Fri, Mar 23, 2018 at 10:03 PM, Denis Magda 
> > wrote:
> > >> > Denis,
> > >> >
> > >> > Thanks for the extensive analysis. There is a vast room for
> > optimizations
> > >> > on the service grid side.
> > >> >
> > >> > Yakov, Sam, Alex G.,
> > >> >
> > >> > How do you like the idea of the usage of discovery protocol for the
> > >> service
> > >> > grid system messages exchange? Any pitfalls?
> > >> >
> > >> >
> > >> > --
> > >> > Denis
> > >> >
> > >> >
> > >> > On Fri, Mar 23, 2018 at 8:01 AM, Denis Mekhanikov <
> > dmekhani...@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> >> Igniters,
> > >> >>
> > >> >> I'd like to start a discussion on Ignite service grid redesign.
> > >> >> We have a number of problems in our current architecture, that have
> > to
> > >> be
> > >> >> addressed.
> > >> >>
> > >> >> Here are the most severe ones:
> > >> >>
> > >> >> One of them is lack of guarantee, that service is successfully
> > deployed
> > >> and
> > >> >> ready for work by the time, when *IgniteService.deploy*()* methods
> > >> return.
> > >> >> Furthermore, if an exception is thrown from *Service.init()
> *method,
> > >> then
> > >> >> the deploying side is not able to receive it, or even understand,
> > that
> > >> >> service is in unusable state.
> > >> >> So, you may end up in such situation, when you deployed a service
> > >> without
> > >> >> receiving any errors, then called a service's method, and hung
> > >> indefinitely
> > >> >> on this invocation.
> > >> >> JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-3392
> > >> >>
> > >> >> Another problem is locking during service deployment on unstable
> > >> topology.
> > >> >> This issue is caused by missing updates in continuous query
> > listeners on
> > >> >> the internal cache.
> > >> >> It is hard to reproduce, but it happens sometimes. We shouldn't
> allow
> > >> such
> > >> >> possibility, that deployment methods hang without saying anything.
> > >> >> JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-6259
> > >> >>
> > >> >> I think, we s

Re: Make TC issues seems to be hung up

[Dmitriy Setrakyan]

Igniters,

Dmitriy Pavlov is working very hard on making sure that we do not have
failing tests in Ignite. Let's help him in this effort. I would like to
encourage the committers responsible for the failing tests to either fix
them or remove them.

Please respond here.

D.

On Tue, Apr 3, 2018 at 5:27 AM, Dmitry Pavlov  wrote:

> Hi Igniters,
>
> Following issues for tests fixing seems have no progress . Mainteiners,
> please address.
>
> https://issues.apache.org/jira/browse/IGNITE-7766
> - Alexei Scherbakov - not responding, if contributor is not interested in
> this test anymore, probably we should remove it from code base?
>
> https://issues.apache.org/jira/browse/IGNITE-7809
> - Ilya Lantukh, Alexey Goncharuk - not responding
>
> https://issues.apache.org/jira/browse/IGNITE-7692
> - Alexey Goncharuk, Vladimir Ozerov - not responding
>
> Folks?
>
> Sincerely,
> Dmitriy Pavlov
>