[MTCGA]: new failures in builds [5780347, 5780334] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New Critical Failure in master Control Utility 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ControlUtility?branch=%3Cdefault%3E
 No changes in the build

 *New Critical Failure in master-nightly MVCC Cache 7 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccCache7?branch=%3Cdefault%3E
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 23:35:59 09-12-2020 

Re: Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

Hi, Andrey!

Users expect DataStorageConfiguration#maxWalArchiveSize to mean that WAL 
archive will not exceed this value, but it is not. 
It seems that to reduce the chance of getting into a situation when we exceed 
WAL archive, it will be lowed when we clean it when switching to a new segment 
than at the end of the checkpoint.
After that, we can think about and make a hard limit on WAL archive, but for 
this will need to solve a few more problems.

09.12.2020, 17:24, "Andrey Gura" :
> Kiriill,
>
> Issue description contains the following:
>
>>  At the moment, WAL archive is cleared at the end of the checkpoint, which 
>> does not seem correct and needs to be moved
>
> Could you please explain why existing behavior is not correct. It
> seems that it is not enough motivation for change.
>
> On Wed, Dec 9, 2020 at 5:05 PM vbm  wrote:
>>  Hi Kirill Tkalenko,
>>
>>  Is there any relation to rate of ingestion of data to ignite ?
>>
>>  We had seen the issue of WAL growing infinitely recently in our K8s cluster.
>>  We were ingesting data at around 2Mbps.
>>  In other clusters where we did not have such a fast ingestion of data, this
>>  issue was not observed.
>>
>>  Regards,
>>  Vishwas
>>
>>  --
>>  Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

Hi, Vishwas!

Speed of uploading data directly associated with the growth of WAL archive.

09.12.2020, 17:05, "vbm" :
> Hi Kirill Tkalenko,
>
> Is there any relation to rate of ingestion of data to ignite ?
>
> We had seen the issue of WAL growing infinitely recently in our K8s cluster.
> We were ingesting data at around 2Mbps.
> In other clusters where we did not have such a fast ingestion of data, this
> issue was not observed.
>
> Regards,
> Vishwas
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: 2.9.1 release scope and dates

[Yaroslav Molochkov]

Guys, hello!

I did some research concerning performance comparison 2.9 and 2.9.1 before
moving any further and starting a vote. It seems that there is no
degradation of any kind.

You can find relevant metrics here

along
with configs used in these tests.


On Thu, Dec 3, 2020 at 12:21 PM Steshin Vladimir  wrote:

>  Yaroslan, helle again.
>
>  Just realized that [1] depends on [4]. [4] is a fix of a reverted
> from 2.9 ticket.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13705
> (edb736dcd8d1d57c875ce7de2b2b2b786d1f8d51)
>
> [4] https://issues.apache.org/jira/browse/IGNITE-13465
> (3db17612d7c433ddfbb404f92eebca6dd2f4fefe)
>
>
>
>  Перенаправленное сообщение 
> Тема:   Re: 2.9.1 release scope and dates
> Дата:   Wed, 2 Dec 2020 13:06:42 +0300
> От: Steshin Vladimir 
> Кому:   dev@ignite.apache.org
>
>
>
> Yaroslav, Hi.
>
>  I propose [1] and [2] to pick up into 2.9.1.
>
>
> [1] is important. It fixes unexpected node failure slipped away from the
> Java test. Belatedly found in integration ducktape tests.
>
>
> [2] just disables soLinger in TcpDiscvoerySPI by default. Suggested in
> 2.10. But the documentation correction has already appeared in the
> documentation [3].
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13705
> (edb736dcd8d1d57c875ce7de2b2b2b786d1f8d51)
>
> [2] https://issues.apache.org/jira/browse/IGNITE-13643
> (cb7448eecf1ae05c2062e24d9c342d8ae9d92149)
>
> [3]
>
> https://ignite.apache.org/docs/latest/clustering/network-configuration#discovery
>
>
> 02.12.2020 12:19, Yaroslav Molochkov пишет:
> > Ivan,
> >
> > thanks, added to the list due to the severity of the issue.
> >
> > On Wed, Dec 2, 2020 at 12:13 PM Ivan Daschinsky
> wrote:
> >
> >> Yaroslav, lets add another bug fix
> >> https://issues.apache.org/jira/browse/IGNITE-13572
> >> It's quite small fix, but bug is quite severe.
> >>
> >> ср, 2 дек. 2020 г. в 10:48, Yaroslav Molochkov:
> >>
> >>> Guys,
> >>>
> >>> can anyone grant me necessary permissions to create a wiki page with
> >>> release info, please?
> >>>
> >>> On Tue, Dec 1, 2020 at 12:58 PM Yaroslav Molochkov <
> >> molochko...@gmail.com>
> >>> wrote:
> >>>
>  Hi!
>  I see it's merged into 2.9.1 branch and master.
> 
>  I guess that's it for the issues, I will run nightly tests again just
> >> to
>  be sure that everything is okay and we will proceed.
> 
>  On Mon, Nov 30, 2020 at 7:31 PM Zhenya Stanilovsky
>    wrote:
> 
> > hello !
> > seems it [1] need to be included too, if it`s not too late, of
> course.
> > May be you can bump reviewer somehow?)
> >
> > [1]https://issues.apache.org/jira/browse/IGNITE-13765
> >
> >> Ivan, it's added, thanks for pointing that out
> >>
> >> On Mon, Nov 30, 2020 at 5:44 PM Ivan Daschinsky <
> >> ivanda...@gmail.com
> > wrote:
> >>> Yaroslav, could you explain why you decided to remove from scope
> >>>   https://issues.apache.org/jira/browse/IGNITE-13699  ?
> >>> This ticket also incorporate few fixes for metrics. Currently
> >> metrics
> > is a
> >>> little bit broken (JoinedCount shows invalid results for example)
> >>>
> >>>
> >>> пн, 30 нояб. 2020 г. в 17:26, Yaroslav Molochkov <
> > molochko...@gmail.com  >:
>  Igniters, hello!
> 
>  First of all, sorry that the release process is taking so long.
> 
>  Secondly, the release build seems stable and no blockers were
> > introduced
>  within that list
>  <
> 
> >>
> https://issues.apache.org/jira/browse/IGNITE-11312?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved
>  (at
>  least on RunAll compared to 2.9)
> 
>  I've also prepared release notes:
> 
>  Ignite Core:
>  * Added support to graceful shutdown for ZookeeperDiscoverySpi
>  * Added System view for binary metadata
>  * Added System view for metastorage items
>  * Added RebalancingPartitionsTotal metrics
>  * Improved checkpoint concurrent behaviour
>  * Fixed critical system error when unregister a JMX bean
>  * Fixed IgniteCache#isClosed() returns false on server node even
> >> if
>  the cache had been closed before
>  * Fixed inability to eagerly remove expired cache entries
>  * Fixed partial index rebuild issue in case indexed cache
> >> contains
>  different datatypes
>  * Fixed local metastorage system view error if unmarshallable
> >>> values
>  present
>  * Fixed deadlock between grid-timeout-worker and a thread
> >> opening a
>  communication connection
>  * Fixed deadlock in IgniteServiceProcessor
>  * Fixed issue when rebalance future might hang in no final 

Re: [DISCUSSION] Modules organization in Ignite 3

[Ilya Kasnacheev]

Hello!

When you do mvn dependencies:copy, you will now end up with
"ignite-core-X.Y.Z.jar", "ignite-indexing-X.Y.Z.jar"
But if we remove "ignite" from artifact name, user will end up with cryptic
"configuration.jar", "raft.jar", etc, etc.

Remember that group name is discarded in file names. Therefore, every
project out there keeps stuff like "spring-" or "hibernate-" in artifact
name.

I also hope that our users will not end up with zillion JARs.

Regards,
-- 
Ilya Kasnacheev


ср, 9 дек. 2020 г. в 16:16, Anton Kalashnikov :

> Hello,
>
> I totally agree that we should start to think about the module
> organization.
> I suggest making the new confluence page where we define new rules on how
> to develop modules.
> In my opinion, at least, the following topics should be covered there(it
> makes sense to discuss every topic separately, not here):
> * In which cases new modules are required
> * The naming of modules, packages
> * Class dependency management ( inversion of control, no more context)
> * Test organization ( module contains only unit, module tests. All
> integration tests are in the extra module)
> * Feature/Module lifecycle - experimental, stable, unstable, deprecated
>
>
> Let's get back to the original topic. I agree with the package naming
> rule:
> if the module name is configuration the package name should be
> org.apache.ignite.configuration and only after that any other subpackages.
> Also, I don't sure that we need ignite- prefix in the module name because
> it doesn't have any extra information:
>
>   org.apache.ignite
>   ignite-configuration
>
> we don't lose anything if convert it to
>
>   org.apache.ignite
>   configuration
>
>
> I also hope that jigsaw can help us somehow with class visibility.
> But if not we can take agreement that for example 'internal' package -
> shouldn't be touched outside of the module -
> of course, using the proper class access level is also the solution(where
> it is possible)
> org.apache.ignite.configuration.Configurator // it has access to the
> internal package but it shouldn't return any class from the internal
> package only from the public one.
> org.apache.ignite.configuration.DynamicProperty //interface
> org.apache.ignite.configuration.internal.properties.IntDynamicProperty
>
>
> User API makes sense only for one end module - ignite(or ignite-core)
> which depends on all other modules
> and doing some integration(adapters) and provide final API for the user.
> So I agree that separated module Ignite-API with zero dependencies will be
> a good solution.
>
> configuration module:
>   Configurator.baseline().enabled() -> DynamicProperties
>
> ignite-api module:
>   BaselineConfiguration.enabled() -> boolean //interface
>
> ignite module:
>   BaselineConfigurationImpl implements BaselineConfiguration{
> Configurator configurator;
> public boolean enabled(){
>return configurator.baseline().enabled().value();
> }
>   }
>
> So maybe my example is not so good. But I want to show that end-user API
> will be defined only in ignite-api
> and you need to adapt it in ignite module which leads to some
> overhead(especially in my example)
>  but it makes development pretty manageable/predictable -
> you can easily implement a new module without any worries that user starts
> to use it.
> It will be available only after making changes in ignite-api.
> The major advantage here is the small size of ignite-api which allows to
> carefully review every change
> which allows keeping ignite API in better quality(I hope at least)
>
> Nikolay, maybe is it better to discuss your question on a separate topic?
> Because looks like it is a pretty discussable topic.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 09.12.2020, 10:31, "Nikolay Izhikov" :
> > Hello, Zhenya, Ivan.
> >
> >>  Hello Nikolay, if i find out introduced features structure in some
> project, i would prefer to choose different one )
> >
> > Many, of the real world users disagree with you.
> > Please, take a look at some examples from widely used projects:
> >
> > Kafka -
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
> > - Stable, Evolving, Unstable
> >
> > Spark -
> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
> > - AlphaComponent, DeveloperApi, Evolving, Experimental, Private,
> Stable, Unstable
> >
> >>  Having officially "unstable" features doesn't sound good for product
> reputation.
> >
> > Can’t agree with you.
> >
> > Forcing ourselves to make perfect API from the first try we just put too
> much pressure on every decision.
> > Every developer making mistakes.
> > The product is evolving and the API too - it’s totally OK.
> >
> > For every new feature time required to be adopted and used in real-world
> production.
> > I believe, slight API changes is totally fine for early adopters.
> > Moreover, I think, that we should 

Re: Move WAL archive cleanup from checkpoint to rollover

[Andrey Gura]

Kiriill,

Issue description contains the following:

> At the moment, WAL archive is cleared at the end of the checkpoint, which 
> does not seem correct and needs to be moved

Could you please explain why existing behavior is not correct. It
seems that it is not enough motivation for change.

On Wed, Dec 9, 2020 at 5:05 PM vbm  wrote:
>
> Hi Kirill Tkalenko,
>
> Is there any relation to rate of ingestion of data to ignite ?
>
> We had seen the issue of WAL growing infinitely recently in our K8s cluster.
> We were ingesting data at around 2Mbps.
> In other clusters where we did not have such a fast ingestion of data, this
> issue was not observed.
>
>
> Regards,
> Vishwas
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Move WAL archive cleanup from checkpoint to rollover

[vbm]

Hi Kirill Tkalenko,

Is there any relation to rate of ingestion of data to ignite ?

We had seen the issue of WAL growing infinitely recently in our K8s cluster.
We were ingesting data at around 2Mbps. 
In other clusters where we did not have such a fast ingestion of data, this
issue was not observed.
 

Regards,
Vishwas



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Issue with custom security plugin and thin clients

[vbm]

Hi Denis,

Any thoughts on the approach mentioned above ?


Regards,
Vishwas



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Modules organization in Ignite 3

[Nikolay Izhikov]

> Nikolay, maybe is it better to discuss your question on a separate topic? 
> Because looks like it is a pretty discussable topic.

Sorry for interrupting original discussion.
For me it was a overlapping topic in the beginning.
Agree that we  we should have separate thread.


> 9 дек. 2020 г., в 16:15, Anton Kalashnikov  написал(а):
> 
> Hello,
> 
> I totally agree that we should start to think about the module organization. 
> I suggest making the new confluence page where we define new rules on how to 
> develop modules. 
> In my opinion, at least, the following topics should be covered there(it 
> makes sense to discuss every topic separately, not here):
> * In which cases new modules are required
> * The naming of modules, packages
> * Class dependency management ( inversion of control, no more context)
> * Test organization ( module contains only unit, module tests. All 
> integration tests are in the extra module)
> * Feature/Module lifecycle - experimental, stable, unstable, deprecated
> 
> 
> Let's get back to the original topic. I agree with the package naming rule: 
> if the module name is configuration the package name should be 
> org.apache.ignite.configuration and only after that any other subpackages.
> Also, I don't sure that we need ignite- prefix in the module name because it 
> doesn't have any extra information:
> 
>  org.apache.ignite 
>  ignite-configuration
> 
> we don't lose anything if convert it to
> 
>  org.apache.ignite 
>  configuration
> 
> 
> I also hope that jigsaw can help us somehow with class visibility. 
> But if not we can take agreement that for example 'internal' package - 
> shouldn't be touched outside of the module - 
> of course, using the proper class access level is also the solution(where it 
> is possible)
> org.apache.ignite.configuration.Configurator // it has access to the internal 
> package but it shouldn't return any class from the internal package only from 
> the public one.
> org.apache.ignite.configuration.DynamicProperty //interface
> org.apache.ignite.configuration.internal.properties.IntDynamicProperty
> 
> 
> User API makes sense only for one end module - ignite(or ignite-core) which 
> depends on all other modules 
> and doing some integration(adapters) and provide final API for the user. 
> So I agree that separated module Ignite-API with zero dependencies will be a 
> good solution.
> 
> configuration module:
>  Configurator.baseline().enabled() -> DynamicProperties
> 
> ignite-api module:
>  BaselineConfiguration.enabled() -> boolean //interface
> 
> ignite module:
>  BaselineConfigurationImpl implements BaselineConfiguration{
> Configurator configurator;
> public boolean enabled(){
>return configurator.baseline().enabled().value();
> }
>  }
> 
> So maybe my example is not so good. But I want to show that end-user API will 
> be defined only in ignite-api 
> and you need to adapt it in ignite module which leads to some 
> overhead(especially in my example)
> but it makes development pretty manageable/predictable - 
> you can easily implement a new module without any worries that user starts to 
> use it. 
> It will be available only after making changes in ignite-api. 
> The major advantage here is the small size of ignite-api which allows to 
> carefully review every change 
> which allows keeping ignite API in better quality(I hope at least)
> 
> Nikolay, maybe is it better to discuss your question on a separate topic? 
> Because looks like it is a pretty discussable topic.
> 
> -- 
> Best regards,
> Anton Kalashnikov
> 
> 
> 
> 09.12.2020, 10:31, "Nikolay Izhikov" :
>> Hello, Zhenya, Ivan.
>> 
>>>  Hello Nikolay, if i find out introduced features structure in some 
>>> project, i would prefer to choose different one )
>> 
>> Many, of the real world users disagree with you.
>> Please, take a look at some examples from widely used projects:
>> 
>> Kafka - 
>> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
>> - Stable, Evolving, Unstable
>> 
>> Spark - 
>> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
>> - AlphaComponent, DeveloperApi, Evolving, Experimental, Private, 
>> Stable, Unstable
>> 
>>>  Having officially "unstable" features doesn't sound good for product 
>>> reputation.
>> 
>> Can’t agree with you.
>> 
>> Forcing ourselves to make perfect API from the first try we just put too 
>> much pressure on every decision.
>> Every developer making mistakes.
>> The product is evolving and the API too - it’s totally OK.
>> 
>> For every new feature time required to be adopted and used in real-world 
>> production.
>> I believe, slight API changes is totally fine for early adopters.
>> Moreover, I think, that we should warn our users that some feature is very 
>> fresh and can have issues.
>> 
>> So, Why Kafka and Spark is good enough to have unstable API and Ignite not? 
>> 

Re: Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

And this property IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE


09.12.2020, 16:28, "ткаленко кирилл" :
> And this property IGNITE_THRESHOLD_WAL_ARCHIVE_SIZE_PERCENTAGE
>
> 09.12.2020, 16:25, "ткаленко кирилл" :
>> Hi, Shiva!
>>
>> I am not aware of such tickets yet. Later I think I can deal with this issue.
>> Now you can try to increase the frequency of checkpoints, the maximum WAL 
>> archive size and try to change the system property 
>> IGNITE_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE.
>> This will not solve the problem, but it may help to delay it.
>>
>> 09.12.2020, 16:15, "shm" :
>>>  Hi Kirill Tkalenko,
>>>  Thanks for your reply!
>>>  Do you know any known issue related to problem I'm facing ? or any ticket
>>>  related to infinite WAL growing issue which can be correlated to my issue?
>>>
>>>  Regards,
>>>  Shiva
>>>
>>>  --
>>>  Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

Hi, Shiva!

I am not aware of such tickets yet. Later I think I can deal with this issue. 
Now you can try to increase the frequency of checkpoints, the maximum WAL 
archive size and try to change the system property 
IGNITE_CHECKPOINT_TRIGGER_ARCHIVE_SIZE_PERCENTAGE.
This will not solve the problem, but it may help to delay it.

09.12.2020, 16:15, "shm" :
> Hi Kirill Tkalenko,
> Thanks for your reply!
> Do you know any known issue related to problem I'm facing ? or any ticket
> related to infinite WAL growing issue which can be correlated to my issue?
>
> Regards,
> Shiva
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Modules organization in Ignite 3

[Anton Kalashnikov]

Hello,

I totally agree that we should start to think about the module organization. 
I suggest making the new confluence page where we define new rules on how to 
develop modules. 
In my opinion, at least, the following topics should be covered there(it makes 
sense to discuss every topic separately, not here):
* In which cases new modules are required
* The naming of modules, packages
* Class dependency management ( inversion of control, no more context)
* Test organization ( module contains only unit, module tests. All integration 
tests are in the extra module)
* Feature/Module lifecycle - experimental, stable, unstable, deprecated


Let's get back to the original topic. I agree with the package naming rule: 
if the module name is configuration the package name should be 
org.apache.ignite.configuration and only after that any other subpackages.
Also, I don't sure that we need ignite- prefix in the module name because it 
doesn't have any extra information:

  org.apache.ignite 
  ignite-configuration

we don't lose anything if convert it to

  org.apache.ignite 
  configuration


I also hope that jigsaw can help us somehow with class visibility. 
But if not we can take agreement that for example 'internal' package - 
shouldn't be touched outside of the module - 
of course, using the proper class access level is also the solution(where it is 
possible)
org.apache.ignite.configuration.Configurator // it has access to the internal 
package but it shouldn't return any class from the internal package only from 
the public one.
org.apache.ignite.configuration.DynamicProperty //interface
org.apache.ignite.configuration.internal.properties.IntDynamicProperty


User API makes sense only for one end module - ignite(or ignite-core) which 
depends on all other modules 
and doing some integration(adapters) and provide final API for the user. 
So I agree that separated module Ignite-API with zero dependencies will be a 
good solution.

configuration module:
  Configurator.baseline().enabled() -> DynamicProperties

ignite-api module:
  BaselineConfiguration.enabled() -> boolean //interface

ignite module:
  BaselineConfigurationImpl implements BaselineConfiguration{
    Configurator configurator;
    public boolean enabled(){
       return configurator.baseline().enabled().value();
    }
  }

So maybe my example is not so good. But I want to show that end-user API will 
be defined only in ignite-api 
and you need to adapt it in ignite module which leads to some 
overhead(especially in my example)
 but it makes development pretty manageable/predictable - 
you can easily implement a new module without any worries that user starts to 
use it. 
It will be available only after making changes in ignite-api. 
The major advantage here is the small size of ignite-api which allows to 
carefully review every change 
which allows keeping ignite API in better quality(I hope at least)

Nikolay, maybe is it better to discuss your question on a separate topic? 
Because looks like it is a pretty discussable topic.

-- 
Best regards,
Anton Kalashnikov



09.12.2020, 10:31, "Nikolay Izhikov" :
> Hello, Zhenya, Ivan.
>
>>  Hello Nikolay, if i find out introduced features structure in some project, 
>> i would prefer to choose different one )
>
> Many, of the real world users disagree with you.
> Please, take a look at some examples from widely used projects:
>
> Kafka - 
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java#L28
> - Stable, Evolving, Unstable
>
> Spark - 
> https://github.com/apache/spark/tree/master/common/tags/src/main/java/org/apache/spark/annotation
> - AlphaComponent, DeveloperApi, Evolving, Experimental, Private, 
> Stable, Unstable
>
>>  Having officially "unstable" features doesn't sound good for product 
>> reputation.
>
> Can’t agree with you.
>
> Forcing ourselves to make perfect API from the first try we just put too much 
> pressure on every decision.
> Every developer making mistakes.
> The product is evolving and the API too - it’s totally OK.
>
> For every new feature time required to be adopted and used in real-world 
> production.
> I believe, slight API changes is totally fine for early adopters.
> Moreover, I think, that we should warn our users that some feature is very 
> fresh and can have issues.
>
> So, Why Kafka and Spark is good enough to have unstable API and Ignite not? :)
>
>>  9 дек. 2020 г., в 10:08, Ivan Bessonov  написал(а):
>>
>>  Conversation shifted into an unintended direction, but I agree.
>>
>>  I think that if API can (or will) be changed then it should be deprecated.
>>  For that
>>  we can introduce @IgniteDeprecated that will contain Ignite version when
>>  API is planned to be removed. Otherwise it's either stable or experimental.
>>  Having officially "unstable" features doesn't sound good for product
>>  reputation.
>>
>>  As for the modularization - I'm all for this idea. If we don't 

Re: Move WAL archive cleanup from checkpoint to rollover

[shm]

Hi Kirill Tkalenko,
Thanks for your reply!
Do you know any known issue related to problem I'm facing ? or any ticket
related to infinite WAL growing issue which can be correlated to my issue?

Regards,
Shiva 




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13832) disco-notifier-worker handles IgniteInterruptedCheckedException incorrectly

[Ivan Bessonov (Jira)]

Ivan Bessonov created IGNITE-13832:
--

 Summary: disco-notifier-worker handles 
IgniteInterruptedCheckedException incorrectly
 Key: IGNITE-13832
 URL: https://issues.apache.org/jira/browse/IGNITE-13832
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


DiscoveryMessageNotifierWorker#body handles InterruptedException correctly but 
if it catches IgniteInterruptedCheckedException, it'll do different logic which 
is incorrect. I believe all InterruptedException should be handled in the same 
way.

 
{code:java}
[org.gridgain:gridgain-compatibility] [2020-04-13 
08:19:15,109][ERROR][disco-notifier-worker-#69754%top2_node_rcv%][root] 
Critical system error detected. Will be handled accordingly to configured 
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class 
o.a.i.IgniteException: Failed to wait for handling disconnect event.]]
[08:19:15]W: [org.gridgain:gridgain-compatibility] class 
org.apache.ignite.IgniteException: Failed to wait for handling disconnect event.
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.awaitDisconnectEvent(GridDiscoveryManager.java:3128)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.access$6400(GridDiscoveryManager.java:2793)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:868)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:519)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2686)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2724)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
java.lang.Thread.run(Thread.java:748)
[08:19:15]W: [org.gridgain:gridgain-compatibility] Caused by: class 
org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted 
while waiting for future to complete.
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:185)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.awaitDisconnectEvent(GridDiscoveryManager.java:3125)
[08:19:15]W: [org.gridgain:gridgain-compatibility]  ... 7 more
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

Hi, Shiva!

Yes, this ticket will only be about moving WAL archive cleanup.
I think further it is possible to solve the problem of WAL archive overflow, 
but before that we need to solve several problems and deal with heuristics.

09.12.2020, 15:02, "shm" :
> Hi Kirill Tkalenko
>
> Is this ticket is just for moving WAL archive clean-up logic from "post
> checkpoint" to FileWriteAheadLogManager#rollOver ? Or is there any plan to
> change the logic which considers Archived WAL segments for clean-up ?
>
> I'm facing a Major issue related to WAL usage and WAL usage growing
> infinitely with heavy write load even maxWalArchiveSize is set.
>
> Ref:
> http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html
>
> http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html
>
> Shiva
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Pull request for minor fix in index page documentation

[Sumit Deshinge]

Great, thanks Denis.

On Tue, Dec 8, 2020 at 11:35 PM Denis Magda  wrote:

> Hi Sumit,
>
> Thanks for catching the issue and preparing a pull-request! The
> pull-request merges changes to your Ignite master branch rather than to the
> Ignite master. Not a bid deal. I see where the problem is. @Nikita Safonov
> , @Viktor Chemodanov ,
> could any of your correct the broken link on the page?
>
> As for the docs contribution process, it's here. There is a step that
> explains how to build the HTML version of the docs locally before pushing
> live:
>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document#HowtoDocument-ContributingtoDocumentation
>
> -
> Denis
>
>
> On Tue, Dec 8, 2020 at 2:21 AM Sumit Deshinge 
> wrote:
>
>> Hi team,
>>
>> I just made a minor change in apache ignite index page documentation.
>> But I am not sure the pull request raised is sufficient or do I need to
>> follow any additional steps. Please let me know/advice on the same.
>> Also is there any way to verify these changes as these are related to
>> documentation being loaded on the ignite website and not related to code?
>> https://github.com/sumitdeshinge/ignite/pull/1
>>
>>
>> --
>> Regards,
>> Sumit Deshinge
>>
>

-- 
Regards,
Sumit Deshinge

Re: Move WAL archive cleanup from checkpoint to rollover

[shm]

Hi Kirill Tkalenko

Is this ticket is just for moving WAL archive clean-up logic from "post
checkpoint" to FileWriteAheadLogManager#rollOver  ? Or is there any plan to
change the logic which considers Archived WAL segments for clean-up ?

I'm facing a Major issue related to WAL usage and WAL usage growing
infinitely with heavy write load even maxWalArchiveSize is set.

Ref:
http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html

http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html

Shiva



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Move WAL archive cleanup from checkpoint to rollover

[ткаленко кирилл]

Hello to all!

At the moment, the archive is cleared at the end of the checkpoint, it seems it 
should happen in FileWriteAheadLogManager. 
I suggest moving it into the FileWriteAheadLogManager#rollOver when the 
DataStorageConfiguration#maxWalArchiveSize is reached.

To do this, I created a ticket 
https://issues.apache.org/jira/browse/IGNITE-13831

[jira] [Created] (IGNITE-13831) Move WAL archive cleanup from checkpoint to rollover

[Kirill Tkalenko (Jira)]

Kirill Tkalenko created IGNITE-13831:


 Summary: Move WAL archive cleanup from checkpoint to rollover
 Key: IGNITE-13831
 URL: https://issues.apache.org/jira/browse/IGNITE-13831
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.10


At the moment, WAL archive is cleared at the end of the checkpoint, which does 
not seem correct and needs to be moved to the *FileWriteAheadLogManager*. 

It is suggested to clear WAL archive in *FileWriteAheadLogManager#rollOver* 
when the *DataStorageConfiguration#maxWalArchiveSize* is reached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Removing MVCC public API

[Maxim Muzafarov]

+1


Also, I want to mention the list of MVCC related opened issues [1]
without any updates for over a year.

[1]  https://s.apache.org/1r5yk

On Wed, 9 Dec 2020 at 10:22, Alexei Scherbakov
 wrote:
>
> +1
>
> ср, 9 дек. 2020 г. в 10:03, Petr Ivanov :
>
> > +1
> >
> >
> > > On 9 Dec 2020, at 09:39, Nikita Amelchev  wrote:
> > >
> > > +1
> > >
> > > ср, 9 дек. 2020 г. в 08:29, ткаленко кирилл :
> > >>
> > >> +1
> > >>
> > >>
> > >> 08.12.2020, 23:47, "Andrey Mashenkov" :
> > >>> +1
> > >>>
> > >>> On Tue, Dec 8, 2020 at 11:22 PM Igor Seliverstov  > >
> > >>> wrote:
> > >>>
> >  +1
> > 
> >  08.12.2020 22:38, Andrey Gura пишет:
> > > +1
> > >
> > > On Tue, Dec 8, 2020 at 10:02 PM Nikolay Izhikov  > >
> >  wrote:
> > >> +1
> > >>
> > >>> 8 дек. 2020 г., в 21:54, Valentin Kulichenko <
> >  valentin.kuliche...@gmail.com> написал(а):
> > >>>
> > >>> +1
> > >>>
> > >>> On Tue, Dec 8, 2020 at 8:31 AM Вячеслав Коптилин <
> >  slava.kopti...@gmail.com>
> > >>> wrote:
> > >>>
> >  Hello Igniters,
> > 
> >  I want to start voting on removing the public API (and eventually
> > all
> >  unused parts) related to the MVCC feature.
> > 
> >  This topic has already been discussed many times (at least, [1],
> > [2])
> >  and
> >  the community has agreed the feature implementation must be
> >  reapproached,
> >  because using coordinator node for transactions ordering and 2pc
> >  protocol
> >  is slow by design and will not scale well. [3]
> > 
> >  Moreover, the current implementation has critical issues [4], not
> >  supported
> >  by the community, and not well tested at all.
> > 
> >  Removing the public API first will allow us to clean up the code
> >  later step
> >  by step without rushing and keep intact useful improvements that
> > are
> >  already in use or can be reused for other parts in the future.
> >  For instance, partition counters implementation is already
> > adapted to
> >  fix
> >  tx caches protocol issues [5].
> > 
> >  The future of MVCC is unclear for now, but, definitely, this
> > feature
> >  is
> >  useful for a lot of user scenarios and can be scheduled for later
> >  Ignite
> >  versions.
> >  Also, the MVCC feature is in an experimental state, so it can be
> >  modified
> >  in any way, I think.
> > 
> >  +1 - to accept removing MVVC feature from public API
> >  0 - don't care either way
> >  -1 - do not accept removing API (explain why)
> > 
> >  The vote will hold for 7 days and will end on Wednesday, December
> >  16th at
> >  19:00 UTC:
> > 
> > 
> > 
> > https://www.timeanddate.com/countdown/generic?iso=20201216T19=1440=cursive
> > 
> >  [1]
> > 
> > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Mark-MVCC-with-IgniteExperimental-td45669.html
> >  [2]
> > 
> > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Disable-MVCC-test-suites-td50416.html
> >  [3]
> > 
> > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Mark-MVCC-with-IgniteExperimental-tp45669p45727.html
> >  [4]
> > 
> > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Mark-MVCC-with-IgniteExperimental-tp45669p45716.html
> >  [5]
> > 
> > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Mark-MVCC-with-IgniteExperimental-tp45669p45714.html
> > 
> >  Thanks,
> >  Slava.
> > 
> > >>>
> > >>> --
> > >>> Best regards,
> > >>> Andrey V. Mashenkov
> > >
> > >
> > >
> > > --
> > > Best wishes,
> > > Amelchev Nikita
> >
> >
>
> --
>
> Best regards,
> Alexei Scherbakov