[MTCGA]: new failures in builds [5418451] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New test failure in master-nightly 
IgfsLocalSecondaryFileSystemDualSyncSelfTest.testAccessAndModificationTimeUpwardsPropagation
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=78576159709943=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 05:28:14 27-06-2020 

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Alex Plehanov]

Guys,

I've created the 2.9 release confluence page [1].
Also, I found that I don't have permission to Team City release tasks. Can
anyone give me such permissions?

[1]: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9

пт, 26 июн. 2020 г. в 13:38, Alexey Zinoviev :

> Igniters
>
> Unfortunately, the ML model export/import feature
>  is under development
> yet.
> I need to delay it before the 2.10 release.
>
>
>
> пт, 26 июн. 2020 г. в 06:50, Alex Plehanov :
>
> > Denis,
> >
> > Yes, I still ready to manage it.
> > Today I will prepare a release page on wiki and will try to go over
> tickets
> > list.
> > Also, I have plans to cut the branch by the end of next week if there are
> > no objections.
> >
> >
> > пт, 26 июн. 2020 г. в 03:48, Denis Magda :
> >
> > > Igniters,
> > >
> > > Are we moving forward with this release? Alex Plehanov, are you still
> > ready
> > > to manage it? It seems like everyone agreed with the timeline you
> > proposed
> > > in the very beginning.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Tue, Jun 16, 2020 at 8:52 AM Denis Magda  wrote:
> > >
> > > > Sergey, Ivan,
> > > >
> > > > Could you please check the questions below? If it's time-consuming to
> > > > rework continuous queries, then the new mode can become available in
> > the
> > > > experimental state and should not let register continuous queries to
> > > avoid
> > > > potential deadlocks. Overall, this design gap in continuous queries
> was
> > > > like a bomb that has just detonated [1]. Anyway, this new
> connectivity
> > > mode
> > > > will be priceless even if you can't use continuous queries with them
> > > > because right now we cannot even start a thick client inside of a
> > > > serverless function.
> > > >
> > > > Alexey Plehanov,
> > > >
> > > > It looks like we can proceed with the release taking your timelines.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-13156
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Wed, Jun 10, 2020 at 4:16 PM Denis Magda 
> wrote:
> > > >
> > > >> Ivan, Sergey,
> > > >>
> > > >> How much effort should we put to resolve the issue with
> > > >> continuous queries? Are you working on that issue actively? Let's
> try
> > to
> > > >> guess what would be the ETA.
> > > >>
> > > >> -
> > > >> Denis
> > > >>
> > > >>
> > > >> On Wed, Jun 10, 2020 at 3:55 AM Ivan Bessonov <
> bessonov...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hello,
> > > >>>
> > > >>> Sorry for the delay. Sergey Chugunov (sergey.chugu...@gmail.com)
> > just
> > > >>> replied
> > > >>> to the main conversation about "communication via discovery" [1].
> We
> > > >>> work on it
> > > >>> together and recently have found one hard-to-fix scenario, detailed
> > > >>> description is
> > > >>> provided in Sergey's reply.
> > > >>>
> > > >>> In short, July 10th looks realistic only if we introduce new
> behavior
> > > in
> > > >>> its current
> > > >>> implementation, with new setting and IgniteExperimental status.
> > Blocker
> > > >>> here is
> > > >>> current implementation of Continuos Query protocol that in some
> cases
> > > >>> (described
> > > >>> at the end) initiates TCP connection right from discovery thread
> > which
> > > >>> obviously
> > > >>> leads to deadlock. We haven't estimated efforts needed to redesign
> of
> > > CQ
> > > >>> protocol
> > > >>> but it is definitely a risk and fixing it isn't feasible with a
> code
> > > >>> freeze at 10th of July.
> > > >>> So my verdict: we can include this new feature in 2.9 scope as
> > > >>> experimental and with
> > > >>> highlighted limitation on CQ usage. Is that OK?
> > > >>>
> > > >>> CQ limitation: server needs to open a communication connection to
> the
> > > >>> client if during
> > > >>> CQ registration client tries to p2p deploy new class not available
> on
> > > >>> server classpath.
> > > >>> In other cases registration of CQ should be fine.
> > > >>>
> > > >>> [1]
> > > >>>
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
> > > >>>
> > > >>> вт, 9 июн. 2020 г. в 19:36, Ivan Rakov :
> > > >>>
> > >  Hi,
> > > 
> > >  Indeed, the tracing feature is almost ready. Discovery,
> > communication
> > >  and
> > >  transactions tracing will be introduced, as well as an option to
> > >  configure
> > >  tracing in runtime. Right now we are working on final performance
> > >  optimizations, but it's very likely that we'll complete this
> > activity
> > >  before the code freeze date.
> > >  Let's include tracing to the 2.9 release scope.
> > > 
> > >  More info:
> > > 
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-48%3A+Tracing
> > >  https://issues.apache.org/jira/browse/IGNITE-13060
> > > 
> > >  --
> > >  Best Regards,
> > >  Ivan Rakov
> > > 
> > >  On Sat, Jun 6, 2020 

Re: Continuous Queries with several remote filter on the same cache

[Denis Magda]

Roman,

The updates are ordered per partition. Let's take this example of an
application updating several records:

put (k1, val1) => mapped to partition_10 => node_A
put (k2, val2) => mapped to partition_5 => node_B
put (k3, val3) => mapped to partition_10 => node_A

It's guaranteed that a continuous query listener will be notified about k1
and k3 updates in this order - k1 first and k3 after. As for the k2 update,
it can arrive at any time (i.e., before k1, after k3 or in the middle).




-
Denis


On Fri, Jun 26, 2020 at 12:58 AM  wrote:

> Hi Denis,
>
> Thanks! Is there some guarantee about the order of the updates? Even when
> we have multiple cache nodes.
>
> Best regards,
> Roman
>
> -Original Message-
> From: Denis Magda 
> Sent: Monday, June 8, 2020 10:20 PM
> To: dev 
> Subject: Re: Continuous Queries with several remote filter on the same
> cache
>
> Roman,
>
> Please check the following methods:
> * CacheContiniousQueryHandler (the filter usage):
>
> https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
> * CacheContinuousQueryManager (the listener execution):
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376
>
> -
> Denis
>
>
> On Sun, Jun 7, 2020 at 12:19 AM  wrote:
>
> > Hi Denis,
> > A big thank you for the answer.
> > Could you please tell me where can I find this logic in the sources.
> > Which package should I look into?
> >
> > -Original Message-
> > From: Denis Magda 
> > Sent: Saturday, June 6, 2020 2:07 AM
> > To: dev 
> > Subject: Re: Continuous Queries with several remote filter on the same
> > cache
> >
> > Hi Roman,
> >
> > Every continuous query is a unique entity that is processed by servers
> > independently. With your example, the server node will execute all 20
> > filters for every cache insert/update operation. The server will
> > notify through local listeners only those clients whose remote filters
> > returned 'true'.
> >
> > -
> > Denis
> >
> >
> > On Thu, Jun 4, 2020 at 8:44 PM  wrote:
> >
> > > Hi Community,
> > >
> > > I ask this question here because I haven't found the answer in the
> > > documentation.
> > >
> > > Could you please clarify how Continuous Queries work? What the
> > > behavior of Continuous Queries if we have several clients with
> > > different Remote Filters on the same cache? For example, if we have:
> > > one server node with cache and we have up to 20 client nodes each of
> > > them will execute Continuous Query on the same cache but with
> > > different Remote Filters. Will each client get the data according to
> > > its remote filter? Or it is supposed to have only one Remote Filter
> > > for all clients and every client should filter data in its local
> > > event
> > listener?
> > > I would be grateful if you send some link which describes the
> > > behavior of Continuous Queries more thoroughly.
> > > Best regards,
> > > Roman
> > >
> >
>

[jira] [Created] (IGNITE-13190) Core defragmentation functions

[Sergey Chugunov (Jira)]

Sergey Chugunov created IGNITE-13190:


 Summary: Core defragmentation functions
 Key: IGNITE-13190
 URL: https://issues.apache.org/jira/browse/IGNITE-13190
 Project: Ignite
  Issue Type: Sub-task
Reporter: Sergey Chugunov


The following set of functions covering defragmentation happy-case needed:
 * Initialization of defragmentation manager when node is started in 
maintenance mode.
 * Information about partition files is gathered by defrag mgr.
 * For each partition file corresponding file of defragmented partition is 
created and initialized.
 * Keys are transferred from old partitions to new partitions.
 * Checkpointer is aware of new partition files and flushes defragmented memory 
to new partition files.

 

No fault-tolerance code nor index defragmentation mappings are needed in this 
task.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13189) Maintenance mode switch and defragmentation process initialization

[Sergey Chugunov (Jira)]

Sergey Chugunov created IGNITE-13189:


 Summary: Maintenance mode switch and defragmentation process 
initialization
 Key: IGNITE-13189
 URL: https://issues.apache.org/jira/browse/IGNITE-13189
 Project: Ignite
  Issue Type: Sub-task
Reporter: Sergey Chugunov


As described in IEP-47 defragmentation is performed when a node enters a 
special mode called maintenance mode.

Discussion on dev-list clarifies algorithm to enter maintenance mode:
 # Special key is written to local metastorage.
 # Node is restarted.
 # Node observes the key on startup and enters maintenance mode.

Node should be fully-functioning in that mode but should not join the rest of 
the cluster and participate in any regular activity like handling cache 
operations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSSION] Add index rebuild time metrics

[ткаленко кирилл]

Hi, Igniters.

I would like to know if it is possible to estimate how much the index rebuild 
will take?

At the moment, I have found the following metrics [1] and [2] and since the 
rebuild is based on caches, I think it would be useful to know how many records 
are processed in indexing. This way we can estimate how long we have to wait 
for the index to be rebuilt by subtracting [3] and how many records are indexed.

I think we should add this metric [4].

Comments, suggestions?

[1] - https://issues.apache.org/jira/browse/IGNITE-12184
[2] - 
org.apache.ignite.internal.processors.cache.CacheGroupMetricsImpl#idxBuildCntPartitionsLeft
[3] - org.apache.ignite.cache.CacheMetrics#getCacheSize
[4] - org.apache.ignite.cache.CacheMetrics#getNumberIndexedKeys

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Alexey Zinoviev]

Igniters

Unfortunately, the ML model export/import feature
 is under development
yet.
I need to delay it before the 2.10 release.



пт, 26 июн. 2020 г. в 06:50, Alex Plehanov :

> Denis,
>
> Yes, I still ready to manage it.
> Today I will prepare a release page on wiki and will try to go over tickets
> list.
> Also, I have plans to cut the branch by the end of next week if there are
> no objections.
>
>
> пт, 26 июн. 2020 г. в 03:48, Denis Magda :
>
> > Igniters,
> >
> > Are we moving forward with this release? Alex Plehanov, are you still
> ready
> > to manage it? It seems like everyone agreed with the timeline you
> proposed
> > in the very beginning.
> >
> > -
> > Denis
> >
> >
> > On Tue, Jun 16, 2020 at 8:52 AM Denis Magda  wrote:
> >
> > > Sergey, Ivan,
> > >
> > > Could you please check the questions below? If it's time-consuming to
> > > rework continuous queries, then the new mode can become available in
> the
> > > experimental state and should not let register continuous queries to
> > avoid
> > > potential deadlocks. Overall, this design gap in continuous queries was
> > > like a bomb that has just detonated [1]. Anyway, this new connectivity
> > mode
> > > will be priceless even if you can't use continuous queries with them
> > > because right now we cannot even start a thick client inside of a
> > > serverless function.
> > >
> > > Alexey Plehanov,
> > >
> > > It looks like we can proceed with the release taking your timelines.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-13156
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Jun 10, 2020 at 4:16 PM Denis Magda  wrote:
> > >
> > >> Ivan, Sergey,
> > >>
> > >> How much effort should we put to resolve the issue with
> > >> continuous queries? Are you working on that issue actively? Let's try
> to
> > >> guess what would be the ETA.
> > >>
> > >> -
> > >> Denis
> > >>
> > >>
> > >> On Wed, Jun 10, 2020 at 3:55 AM Ivan Bessonov 
> > >> wrote:
> > >>
> > >>> Hello,
> > >>>
> > >>> Sorry for the delay. Sergey Chugunov (sergey.chugu...@gmail.com)
> just
> > >>> replied
> > >>> to the main conversation about "communication via discovery" [1]. We
> > >>> work on it
> > >>> together and recently have found one hard-to-fix scenario, detailed
> > >>> description is
> > >>> provided in Sergey's reply.
> > >>>
> > >>> In short, July 10th looks realistic only if we introduce new behavior
> > in
> > >>> its current
> > >>> implementation, with new setting and IgniteExperimental status.
> Blocker
> > >>> here is
> > >>> current implementation of Continuos Query protocol that in some cases
> > >>> (described
> > >>> at the end) initiates TCP connection right from discovery thread
> which
> > >>> obviously
> > >>> leads to deadlock. We haven't estimated efforts needed to redesign of
> > CQ
> > >>> protocol
> > >>> but it is definitely a risk and fixing it isn't feasible with a code
> > >>> freeze at 10th of July.
> > >>> So my verdict: we can include this new feature in 2.9 scope as
> > >>> experimental and with
> > >>> highlighted limitation on CQ usage. Is that OK?
> > >>>
> > >>> CQ limitation: server needs to open a communication connection to the
> > >>> client if during
> > >>> CQ registration client tries to p2p deploy new class not available on
> > >>> server classpath.
> > >>> In other cases registration of CQ should be fine.
> > >>>
> > >>> [1]
> > >>>
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
> > >>>
> > >>> вт, 9 июн. 2020 г. в 19:36, Ivan Rakov :
> > >>>
> >  Hi,
> > 
> >  Indeed, the tracing feature is almost ready. Discovery,
> communication
> >  and
> >  transactions tracing will be introduced, as well as an option to
> >  configure
> >  tracing in runtime. Right now we are working on final performance
> >  optimizations, but it's very likely that we'll complete this
> activity
> >  before the code freeze date.
> >  Let's include tracing to the 2.9 release scope.
> > 
> >  More info:
> > 
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-48%3A+Tracing
> >  https://issues.apache.org/jira/browse/IGNITE-13060
> > 
> >  --
> >  Best Regards,
> >  Ivan Rakov
> > 
> >  On Sat, Jun 6, 2020 at 4:30 PM Denis Magda 
> wrote:
> > 
> >  > Hi folks,
> >  >
> >  > The timelines proposed by Alex Plekhanov sounds reasonable to me.
> > I'd
> >  like
> >  > only to hear inputs of @Ivan Rakov , who is
> >  about to
> >  > finish with the tracing support, and @Ivan Bessonov
> >  > , who is fixing a serious limitation for
> K8
> >  > deployments [1]. Most likely, both features will be ready by the
> > code
> >  > freeze date (July 10), but the guys should know it better.
> >  >
> >  > [1]
> >  >
> > 
> >
> 

RE: Continuous Queries with several remote filter on the same cache

[Roman.Koriakov]

Hi Denis,

Thanks! Is there some guarantee about the order of the updates? Even when we 
have multiple cache nodes.

Best regards,
Roman

-Original Message-
From: Denis Magda  
Sent: Monday, June 8, 2020 10:20 PM
To: dev 
Subject: Re: Continuous Queries with several remote filter on the same cache

Roman,

Please check the following methods:
* CacheContiniousQueryHandler (the filter usage):
https://github.com/apache/ignite/blob/6955ac291352dd67c1f84a006cda512ee54f38bb/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryHandler.java#L994
* CacheContinuousQueryManager (the listener execution):
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryManager.java#L376

-
Denis


On Sun, Jun 7, 2020 at 12:19 AM  wrote:

> Hi Denis,
> A big thank you for the answer.
> Could you please tell me where can I find this logic in the sources. 
> Which package should I look into?
>
> -Original Message-
> From: Denis Magda 
> Sent: Saturday, June 6, 2020 2:07 AM
> To: dev 
> Subject: Re: Continuous Queries with several remote filter on the same 
> cache
>
> Hi Roman,
>
> Every continuous query is a unique entity that is processed by servers 
> independently. With your example, the server node will execute all 20 
> filters for every cache insert/update operation. The server will 
> notify through local listeners only those clients whose remote filters 
> returned 'true'.
>
> -
> Denis
>
>
> On Thu, Jun 4, 2020 at 8:44 PM  wrote:
>
> > Hi Community,
> >
> > I ask this question here because I haven't found the answer in the 
> > documentation.
> >
> > Could you please clarify how Continuous Queries work? What the 
> > behavior of Continuous Queries if we have several clients with 
> > different Remote Filters on the same cache? For example, if we have:
> > one server node with cache and we have up to 20 client nodes each of 
> > them will execute Continuous Query on the same cache but with 
> > different Remote Filters. Will each client get the data according to 
> > its remote filter? Or it is supposed to have only one Remote Filter 
> > for all clients and every client should filter data in its local 
> > event
> listener?
> > I would be grateful if you send some link which describes the 
> > behavior of Continuous Queries more thoroughly.
> > Best regards,
> > Roman
> >
>

[jira] [Created] (IGNITE-13188) In SQLBindParameter if StrLen_or_IndPtr is NULL then default value should be SQL_NTS

[Abhay (Jira)]

Abhay created IGNITE-13188:
--

 Summary: In SQLBindParameter if StrLen_or_IndPtr is NULL then 
default value should be SQL_NTS
 Key: IGNITE-13188
 URL: https://issues.apache.org/jira/browse/IGNITE-13188
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.8.1, 2.7.6
Reporter: Abhay


We had a issue in integrating Asterisk VOIP server with Ignite and found that 
in app/application_data_buffer.cpp 

SqlLen ApplicationDataBuffer::GetInputSize() const

this line 

return len ? *len : SQL_DEFAULT_PARAM;

should be replaced with 

return len ? *len : SQL_NTS;

As per RFC 

[https://docs.microsoft.com/en-us/sql/odbc/reference/syntax/sqlbindparameter-function?view=sql-server-ver15]

If StrLen_or_IndPtr is NULL then it should be treated as null terminated string 
and so SQL_NTS should be the value .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)