[jira] [Created] (IGNITE-12684) Optimize count of threads for rebuild indexes
Kirill Tkalenko created IGNITE-12684:
Summary: Optimize count of threads for rebuild indexes
Key: IGNITE-12684
URL: https://issues.apache.org/jira/browse/IGNITE-12684
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.9
We were recently implemented the new property {{INDEX_REBUILDING_PARALLELISM}}
(by issue IGNITE-11979).
That fix do possible to rebuild indexes, after restart without index.bin or
restore from snapshot to another topology, in parallel threads.
But now the process of rebuild can to consume a lot of threads, which limited
on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in public
pool).
For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will be
more then 1.5К threads.
Look at the method, it will create dedicate thread on each invocation
`org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vyacheslav, >>> I am talking about MIXED cluster with persistent cache and *in-memory* cache which is backed by *3-rd party persistence*. We do not know what exactly does *3-rd party persistence*. It may store only specific filtered data, small part of data. I think it is out of responsibility of the persistence. Such caches might be considered as in-memory only. >>> That is why I do not like to expose such functionality through JMX. But it is exposed also in CLI and REST. Just various call types. Why hide it from JMX? Or why it is supposed to act differently? If CLI (and REST) requires forced deactivation, why JMX would act in other way at the same conditions? пт, 14 февр. 2020 г. в 18:40, Вячеслав Коптилин : > Hi Vladimir, > > > if you have only persistent caches, no warning/confirmation is supposed > at all. > I am talking about MIXED cluster with persistent cache and *in-memory* > cache which is backed by *3-rd party persistence*. > > > I’m afraid this won’t stop anyone from using old deprecated > IgniteMXBean#active(boolean) > That is why I do not like to expose such functionality through JMX. > > Thanks, > S. > > пт, 14 февр. 2020 г. в 18:02, Vladimir Steshin : > > > Vyacheslav, > > > > >>> Let's assume that I have a mixed cluster with persistent cache and > > in-memory cache which is backed by 3-rd party persistence. I see no > reason > > to throw an exception in that case at least. > > > > > > > > if you have only persistent caches, no warning/confirmation is supposed > at > > all. > > > > > > > > >>> Is it possible to > > add new methods as follows: activateCluster()/deactivateCluster() and > > deprecate IgniteMXBean#active(boolean)? > > > > > > > > I’m afraid this won’t stop anyone from using old deprecated > > IgniteMXBean#active(boolean). > > It is quite obvious to execute through JMX despite it is deprecated. > > > > пт, 14 февр. 2020 г. в 17:36, Вячеслав Коптилин < > slava.kopti...@gmail.com > > >: > > > > > Hello Nikolay, > > > > > > > Should public java API continue to silently clear in-memory caches? > > > Let's assume that I have a mixed cluster with persistent cache and > > > in-memory cache which is backed by 3-rd party persistence. I see no > > reason > > > to throw an exception in that case at least. > > > Anyway, this fact should be clearly stated in the Javadoc and > > documentation > > > of course. > > > > > > > What is your suggestion for the API? > > > I think we are talking about JMX methods. Am I correct? Is it possible > to > > > add new methods as follows: activateCluster()/deactivateCluster() and > > > deprecate IgniteMXBean#active(boolean)? > > > Does this make sense? Am I missing something? > > > > > > Thanks, > > > S. > > > > > > пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov : > > > > > > > Vyacheslav. > > > > > > > > What is your suggestion for the API? > > > > > > > > Single implementation for both Ignite#active(boolean) and > > > > IgniteMXBean#active(boolean) > > > > Should public java API continue to silently clears in-memory caches? > > > > > > > > > > > > > 14 февр. 2020 г., в 15:56, Вячеслав Коптилин < > > slava.kopti...@gmail.com > > > > > > > > написал(а): > > > > > > > > > > Hello Vladimir, > > > > > > > > > >> adding a new method with force flag means old methods change their > > > > > behavior: > > > > > I don't think that changing the behavior of public API is the right > > > way. > > > > > Moreover, I agree with Alex that there is no need to introduce a > > > > > "confirmation" flag to the java API. > > > > > > > > > > Thanks, > > > > > S. > > > > > > > > > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin >: > > > > > > > > > >> Alexey, adding a new method with force flag means old methods > change > > > > their > > > > >> behavior: they are considered as executed without ‘force‘ flag and > > can > > > > fail > > > > >> to prevent data loss. Ignite and IgniteMXBean are different > > > interfaces. > > > > >> Unfortunately, they have same method > > > > >> > > > > >> void active(boolean active) > > > > >> > > > > >> When executed as IgniteMXBean it should fail if user can lose > data. > > > When > > > > >> executed from code via interface Ignite probably not. To solve > this > > I > > > > >> suggest to add ‘force’ flag for every deactivation mode: > > CLI/JMX/REST > > > > and > > > > >> other API. > > > > >> > > > > >> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com > > > > >>> : > > > > >> > > > > >>> Igniters, > > > > >>> > > > > >>> Do we really need the confirmation flag on the public API? I > > > absolutely > > > > >>> agree on the CLI and MXBean, but what is the reason for the flag > in > > > the > > > > >>> API? It will be specified at the compile time anyway and does not > > > > prevent > > > > >>> any user error. > > > > >>> From the implementation point of view I see no contradiction - we > > can > > > > add > > > > >>> the new method to the MXBean, but nothing forces us to add it to > > > Ignite >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hi Vladimir, > if you have only persistent caches, no warning/confirmation is supposed at all. I am talking about MIXED cluster with persistent cache and *in-memory* cache which is backed by *3-rd party persistence*. > I’m afraid this won’t stop anyone from using old deprecated IgniteMXBean#active(boolean) That is why I do not like to expose such functionality through JMX. Thanks, S. пт, 14 февр. 2020 г. в 18:02, Vladimir Steshin : > Vyacheslav, > > >>> Let's assume that I have a mixed cluster with persistent cache and > in-memory cache which is backed by 3-rd party persistence. I see no reason > to throw an exception in that case at least. > > > > if you have only persistent caches, no warning/confirmation is supposed at > all. > > > > >>> Is it possible to > add new methods as follows: activateCluster()/deactivateCluster() and > deprecate IgniteMXBean#active(boolean)? > > > > I’m afraid this won’t stop anyone from using old deprecated > IgniteMXBean#active(boolean). > It is quite obvious to execute through JMX despite it is deprecated. > > пт, 14 февр. 2020 г. в 17:36, Вячеслав Коптилин >: > > > Hello Nikolay, > > > > > Should public java API continue to silently clear in-memory caches? > > Let's assume that I have a mixed cluster with persistent cache and > > in-memory cache which is backed by 3-rd party persistence. I see no > reason > > to throw an exception in that case at least. > > Anyway, this fact should be clearly stated in the Javadoc and > documentation > > of course. > > > > > What is your suggestion for the API? > > I think we are talking about JMX methods. Am I correct? Is it possible to > > add new methods as follows: activateCluster()/deactivateCluster() and > > deprecate IgniteMXBean#active(boolean)? > > Does this make sense? Am I missing something? > > > > Thanks, > > S. > > > > пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov : > > > > > Vyacheslav. > > > > > > What is your suggestion for the API? > > > > > > Single implementation for both Ignite#active(boolean) and > > > IgniteMXBean#active(boolean) > > > Should public java API continue to silently clears in-memory caches? > > > > > > > > > > 14 февр. 2020 г., в 15:56, Вячеслав Коптилин < > slava.kopti...@gmail.com > > > > > > написал(а): > > > > > > > > Hello Vladimir, > > > > > > > >> adding a new method with force flag means old methods change their > > > > behavior: > > > > I don't think that changing the behavior of public API is the right > > way. > > > > Moreover, I agree with Alex that there is no need to introduce a > > > > "confirmation" flag to the java API. > > > > > > > > Thanks, > > > > S. > > > > > > > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > > > > > > > >> Alexey, adding a new method with force flag means old methods change > > > their > > > >> behavior: they are considered as executed without ‘force‘ flag and > can > > > fail > > > >> to prevent data loss. Ignite and IgniteMXBean are different > > interfaces. > > > >> Unfortunately, they have same method > > > >> > > > >> void active(boolean active) > > > >> > > > >> When executed as IgniteMXBean it should fail if user can lose data. > > When > > > >> executed from code via interface Ignite probably not. To solve this > I > > > >> suggest to add ‘force’ flag for every deactivation mode: > CLI/JMX/REST > > > and > > > >> other API. > > > >> > > > >> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > >>> : > > > >> > > > >>> Igniters, > > > >>> > > > >>> Do we really need the confirmation flag on the public API? I > > absolutely > > > >>> agree on the CLI and MXBean, but what is the reason for the flag in > > the > > > >>> API? It will be specified at the compile time anyway and does not > > > prevent > > > >>> any user error. > > > >>> From the implementation point of view I see no contradiction - we > can > > > add > > > >>> the new method to the MXBean, but nothing forces us to add it to > > Ignite > > > >>> interface - those interfaces are not related. > > > >>> > > > >> > > > > > > > > >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vyacheslav, >>> Let's assume that I have a mixed cluster with persistent cache and in-memory cache which is backed by 3-rd party persistence. I see no reason to throw an exception in that case at least. if you have only persistent caches, no warning/confirmation is supposed at all. >>> Is it possible to add new methods as follows: activateCluster()/deactivateCluster() and deprecate IgniteMXBean#active(boolean)? I’m afraid this won’t stop anyone from using old deprecated IgniteMXBean#active(boolean). It is quite obvious to execute through JMX despite it is deprecated. пт, 14 февр. 2020 г. в 17:36, Вячеслав Коптилин : > Hello Nikolay, > > > Should public java API continue to silently clear in-memory caches? > Let's assume that I have a mixed cluster with persistent cache and > in-memory cache which is backed by 3-rd party persistence. I see no reason > to throw an exception in that case at least. > Anyway, this fact should be clearly stated in the Javadoc and documentation > of course. > > > What is your suggestion for the API? > I think we are talking about JMX methods. Am I correct? Is it possible to > add new methods as follows: activateCluster()/deactivateCluster() and > deprecate IgniteMXBean#active(boolean)? > Does this make sense? Am I missing something? > > Thanks, > S. > > пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov : > > > Vyacheslav. > > > > What is your suggestion for the API? > > > > Single implementation for both Ignite#active(boolean) and > > IgniteMXBean#active(boolean) > > Should public java API continue to silently clears in-memory caches? > > > > > > > 14 февр. 2020 г., в 15:56, Вячеслав Коптилин > > > написал(а): > > > > > > Hello Vladimir, > > > > > >> adding a new method with force flag means old methods change their > > > behavior: > > > I don't think that changing the behavior of public API is the right > way. > > > Moreover, I agree with Alex that there is no need to introduce a > > > "confirmation" flag to the java API. > > > > > > Thanks, > > > S. > > > > > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > > > > > >> Alexey, adding a new method with force flag means old methods change > > their > > >> behavior: they are considered as executed without ‘force‘ flag and can > > fail > > >> to prevent data loss. Ignite and IgniteMXBean are different > interfaces. > > >> Unfortunately, they have same method > > >> > > >> void active(boolean active) > > >> > > >> When executed as IgniteMXBean it should fail if user can lose data. > When > > >> executed from code via interface Ignite probably not. To solve this I > > >> suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST > > and > > >> other API. > > >> > > >> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > > alexey.goncha...@gmail.com > > >>> : > > >> > > >>> Igniters, > > >>> > > >>> Do we really need the confirmation flag on the public API? I > absolutely > > >>> agree on the CLI and MXBean, but what is the reason for the flag in > the > > >>> API? It will be specified at the compile time anyway and does not > > prevent > > >>> any user error. > > >>> From the implementation point of view I see no contradiction - we can > > add > > >>> the new method to the MXBean, but nothing forces us to add it to > Ignite > > >>> interface - those interfaces are not related. > > >>> > > >> > > > > >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Soory.
I forgot two more methods
IgniteMXBean {
@Deprecated
public String clusterState();
public void clusterState(String state);
}
> 14 февр. 2020 г., в 17:50, Nikolay Izhikov
> написал(а):
>
> Vyacheslav.
>
>
> Let’s see what we got for now:
>
> ```
> Ignite {
>
>@Deprecated
>public boolean active();
>
>@Deprecated
>public void active(boolean active);
> }
>
> IgniteMXBean {
>@Deprecated
>public boolean active();
>
>public void active(boolean active); //Please, note, this method is not
> deprecated. It has the same implementation as Ignite#active(boolean)
> }
>
> IgniteCluster {
>@Deprecated
>public boolean active();
>
>@Deprecated
>public void active(boolean active);
>
>public ClusterState state();
>
>public void state(ClusterState newState) throws IgniteException;
> }
> ```
>
> This issue with the IgniteMXBean#active(boolean) - it has the same
> implementation(IgniteKernal#active) as Ignite#active.
> We can’t change IgniteMXBean#active without changing Ignite#active.
>
> Moreover, for now, we already have an API mess for cluster activation.
> If we introduce different behavior for the different interfaces(java, JMX) it
> will confuse users even more.
>
>> I see no reason to throw an exception in that case at least.
>
> Users(and even Ignite PMCs) doesn’t expect that in-memory cache will be
> cleared on deactivation.
> I think we should notify users about this detail in very clear fashion -
> require explicit confirmation for the operation.
>
>> Anyway, this fact should be clearly stated in the Javadoc and documentation
>> of course.
>
> This is default requirement for the improvement.
> It will be done.
>
>
>
>> 14 февр. 2020 г., в 17:36, Вячеслав Коптилин
>> написал(а):
>>
>> Hello Nikolay,
>>
>>> Should public java API continue to silently clear in-memory caches?
>> Let's assume that I have a mixed cluster with persistent cache and
>> in-memory cache which is backed by 3-rd party persistence. I see no reason
>> to throw an exception in that case at least.
>> Anyway, this fact should be clearly stated in the Javadoc and documentation
>> of course.
>>
>>> What is your suggestion for the API?
>> I think we are talking about JMX methods. Am I correct? Is it possible to
>> add new methods as follows: activateCluster()/deactivateCluster() and
>> deprecate IgniteMXBean#active(boolean)?
>> Does this make sense? Am I missing something?
>>
>> Thanks,
>> S.
>>
>> пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov :
>>
>>> Vyacheslav.
>>>
>>> What is your suggestion for the API?
>>>
>>> Single implementation for both Ignite#active(boolean) and
>>> IgniteMXBean#active(boolean)
>>> Should public java API continue to silently clears in-memory caches?
>>>
>>>
14 февр. 2020 г., в 15:56, Вячеслав Коптилин
>>> написал(а):
Hello Vladimir,
> adding a new method with force flag means old methods change their
behavior:
I don't think that changing the behavior of public API is the right way.
Moreover, I agree with Alex that there is no need to introduce a
"confirmation" flag to the java API.
Thanks,
S.
пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin :
> Alexey, adding a new method with force flag means old methods change
>>> their
> behavior: they are considered as executed without ‘force‘ flag and can
>>> fail
> to prevent data loss. Ignite and IgniteMXBean are different interfaces.
> Unfortunately, they have same method
>
> void active(boolean active)
>
> When executed as IgniteMXBean it should fail if user can lose data. When
> executed from code via interface Ignite probably not. To solve this I
> suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST
>>> and
> other API.
>
> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk <
>>> alexey.goncha...@gmail.com
>> :
>
>> Igniters,
>>
>> Do we really need the confirmation flag on the public API? I absolutely
>> agree on the CLI and MXBean, but what is the reason for the flag in the
>> API? It will be specified at the compile time anyway and does not
>>> prevent
>> any user error.
>> From the implementation point of view I see no contradiction - we can
>>> add
>> the new method to the MXBean, but nothing forces us to add it to Ignite
>> interface - those interfaces are not related.
>>
>
>>>
>>>
>
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vyacheslav.
Let’s see what we got for now:
```
Ignite {
@Deprecated
public boolean active();
@Deprecated
public void active(boolean active);
}
IgniteMXBean {
@Deprecated
public boolean active();
public void active(boolean active); //Please, note, this method is not
deprecated. It has the same implementation as Ignite#active(boolean)
}
IgniteCluster {
@Deprecated
public boolean active();
@Deprecated
public void active(boolean active);
public ClusterState state();
public void state(ClusterState newState) throws IgniteException;
}
```
This issue with the IgniteMXBean#active(boolean) - it has the same
implementation(IgniteKernal#active) as Ignite#active.
We can’t change IgniteMXBean#active without changing Ignite#active.
Moreover, for now, we already have an API mess for cluster activation.
If we introduce different behavior for the different interfaces(java, JMX) it
will confuse users even more.
> I see no reason to throw an exception in that case at least.
Users(and even Ignite PMCs) doesn’t expect that in-memory cache will be cleared
on deactivation.
I think we should notify users about this detail in very clear fashion -
require explicit confirmation for the operation.
> Anyway, this fact should be clearly stated in the Javadoc and documentation
> of course.
This is default requirement for the improvement.
It will be done.
> 14 февр. 2020 г., в 17:36, Вячеслав Коптилин
> написал(а):
>
> Hello Nikolay,
>
>> Should public java API continue to silently clear in-memory caches?
> Let's assume that I have a mixed cluster with persistent cache and
> in-memory cache which is backed by 3-rd party persistence. I see no reason
> to throw an exception in that case at least.
> Anyway, this fact should be clearly stated in the Javadoc and documentation
> of course.
>
>> What is your suggestion for the API?
> I think we are talking about JMX methods. Am I correct? Is it possible to
> add new methods as follows: activateCluster()/deactivateCluster() and
> deprecate IgniteMXBean#active(boolean)?
> Does this make sense? Am I missing something?
>
> Thanks,
> S.
>
> пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov :
>
>> Vyacheslav.
>>
>> What is your suggestion for the API?
>>
>> Single implementation for both Ignite#active(boolean) and
>> IgniteMXBean#active(boolean)
>> Should public java API continue to silently clears in-memory caches?
>>
>>
>>> 14 февр. 2020 г., в 15:56, Вячеслав Коптилин
>> написал(а):
>>>
>>> Hello Vladimir,
>>>
adding a new method with force flag means old methods change their
>>> behavior:
>>> I don't think that changing the behavior of public API is the right way.
>>> Moreover, I agree with Alex that there is no need to introduce a
>>> "confirmation" flag to the java API.
>>>
>>> Thanks,
>>> S.
>>>
>>> пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin :
>>>
Alexey, adding a new method with force flag means old methods change
>> their
behavior: they are considered as executed without ‘force‘ flag and can
>> fail
to prevent data loss. Ignite and IgniteMXBean are different interfaces.
Unfortunately, they have same method
void active(boolean active)
When executed as IgniteMXBean it should fail if user can lose data. When
executed from code via interface Ignite probably not. To solve this I
suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST
>> and
other API.
пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk <
>> alexey.goncha...@gmail.com
> :
> Igniters,
>
> Do we really need the confirmation flag on the public API? I absolutely
> agree on the CLI and MXBean, but what is the reason for the flag in the
> API? It will be specified at the compile time anyway and does not
>> prevent
> any user error.
> From the implementation point of view I see no contradiction - we can
>> add
> the new method to the MXBean, but nothing forces us to add it to Ignite
> interface - those interfaces are not related.
>
>>
>>
Re: [DISCUSS] Relevance of pessimistic tx log properties in TransactionConfiguration
No ) The reasons is serialization of this field. We can mark it as deprecated and remove in 3.0. On Thu, Feb 13, 2020 at 6:08 PM Alexei Scherbakov wrote: > > I think yes. > > чт, 13 февр. 2020 г. в 16:52, Ivan Pavlukhin : > > > Hi, > > > > It seems I found some more unused configuration properties: > > * TransactionConfiguration.PessimisticTxLogSize > > * TransactionConfiguration.PessimisticTxLogLinger > > > > Can we deprecate/remove them? > > > > Best regards, > > Ivan Pavlukhin > > > > > -- > > Best regards, > Alexei Scherbakov
Re: [DISCUSS] Relevance of CacheConfiguration.DefaultLockTimeout
I couldn't find any usages too. But value if this property is serialized in some places and we can't remove this due to a backward compatibility. I think we can safely remove it on 3.0. But for now it is better to mark it as deprecated. On Thu, Feb 13, 2020 at 5:42 PM Alexey Goncharuk wrote: > > Ivan, > > Nothing from my side as well - looks like the property is not used.
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hello Nikolay, > Should public java API continue to silently clear in-memory caches? Let's assume that I have a mixed cluster with persistent cache and in-memory cache which is backed by 3-rd party persistence. I see no reason to throw an exception in that case at least. Anyway, this fact should be clearly stated in the Javadoc and documentation of course. > What is your suggestion for the API? I think we are talking about JMX methods. Am I correct? Is it possible to add new methods as follows: activateCluster()/deactivateCluster() and deprecate IgniteMXBean#active(boolean)? Does this make sense? Am I missing something? Thanks, S. пт, 14 февр. 2020 г. в 16:17, Nikolay Izhikov : > Vyacheslav. > > What is your suggestion for the API? > > Single implementation for both Ignite#active(boolean) and > IgniteMXBean#active(boolean) > Should public java API continue to silently clears in-memory caches? > > > > 14 февр. 2020 г., в 15:56, Вячеслав Коптилин > написал(а): > > > > Hello Vladimir, > > > >> adding a new method with force flag means old methods change their > > behavior: > > I don't think that changing the behavior of public API is the right way. > > Moreover, I agree with Alex that there is no need to introduce a > > "confirmation" flag to the java API. > > > > Thanks, > > S. > > > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > > > >> Alexey, adding a new method with force flag means old methods change > their > >> behavior: they are considered as executed without ‘force‘ flag and can > fail > >> to prevent data loss. Ignite and IgniteMXBean are different interfaces. > >> Unfortunately, they have same method > >> > >> void active(boolean active) > >> > >> When executed as IgniteMXBean it should fail if user can lose data. When > >> executed from code via interface Ignite probably not. To solve this I > >> suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST > and > >> other API. > >> > >> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > alexey.goncha...@gmail.com > >>> : > >> > >>> Igniters, > >>> > >>> Do we really need the confirmation flag on the public API? I absolutely > >>> agree on the CLI and MXBean, but what is the reason for the flag in the > >>> API? It will be specified at the compile time anyway and does not > prevent > >>> any user error. > >>> From the implementation point of view I see no contradiction - we can > add > >>> the new method to the MXBean, but nothing forces us to add it to Ignite > >>> interface - those interfaces are not related. > >>> > >> > >
Re: IGNITE-10698 Task
Lev, it's enough to comment corresponding JIRA ticket. It already contains all needed info (links to PR, TC bot, etc) and also all changes to the ticket will be sent to the dev list. On Fri, Feb 14, 2020 at 1:31 PM Лев Киселев wrote: > > Hello everyone, > I've fixed codestyle issues in my branch and got "No possible blockers" in > TC bot, so PR is currently ready for review. > PR github link: https://github.com/apache/ignite/pull/7394 > Jira task link: https://issues.apache.org/jira/browse/IGNITE-10698 > TeamcityBot link: > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/7394/head&action=Latest
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
* The pr: https://github.com/apache/ignite/pull/7358 One can see the solution closer. пт, 14 февр. 2020 г. в 15:57, Вячеслав Коптилин : > Hello Vladimir, > > > adding a new method with force flag means old methods change their > behavior: > I don't think that changing the behavior of public API is the right way. > Moreover, I agree with Alex that there is no need to introduce a > "confirmation" flag to the java API. > > Thanks, > S. > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > > > Alexey, adding a new method with force flag means old methods change > their > > behavior: they are considered as executed without ‘force‘ flag and can > fail > > to prevent data loss. Ignite and IgniteMXBean are different interfaces. > > Unfortunately, they have same method > > > > void active(boolean active) > > > > When executed as IgniteMXBean it should fail if user can lose data. When > > executed from code via interface Ignite probably not. To solve this I > > suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST and > > other API. > > > > пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > Igniters, > > > > > > Do we really need the confirmation flag on the public API? I absolutely > > > agree on the CLI and MXBean, but what is the reason for the flag in the > > > API? It will be specified at the compile time anyway and does not > prevent > > > any user error. > > > From the implementation point of view I see no contradiction - we can > add > > > the new method to the MXBean, but nothing forces us to add it to Ignite > > > interface - those interfaces are not related. > > > > > >
[jira] [Created] (IGNITE-12683) BPlusTreeReuseListPageMemoryImplTest.testIterateConcurrentPutRemove_2 fails with timeout on TC
Pavel Pereslegin created IGNITE-12683: - Summary: BPlusTreeReuseListPageMemoryImplTest.testIterateConcurrentPutRemove_2 fails with timeout on TC Key: IGNITE-12683 URL: https://issues.apache.org/jira/browse/IGNITE-12683 Project: Ignite Issue Type: Bug Reporter: Pavel Pereslegin Assignee: Pavel Pereslegin Attachments: image-2020-02-14-16-08-54-236.png, image-2020-02-14-16-09-16-401.png, image-2020-02-14-16-09-32-405.png BPlusTreeReuseListPageMemoryImplTest.testIterateConcurrentPutRemove_2 flaky fails on master branch with a high failure rate: [https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-7895536196794411367&tab=testDetails&branch_IgniteTests24Java8=%3Cdefault%3E] It seems that sometimes agent is not fast enough to execute this test in 5 mins. !image-2020-02-14-16-09-32-405.png|width=1149,height=201! Also, I compared master with ignite-2.7.6 (just manually set MAX_PER_PAGE=2) and found locally about ~10% performance difference in test execution. With disabled page lock tracker (IGNITE_PAGE_LOCK_TRACKER_TYPE=-1) there is no difference in test execution. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Вячеслав, Hi. Even if previous behavior is controversial? It allows to erase data suddenly. I would suggest to slightly change API of IgniteMXBean. It would solve the problem of single implementation IgniteKernal#active(boolean). But changing of API is prohibited. Another solution is to extract implementation of IgniteMXBean from IgniteKernal. I believe this approach is too bulky. How often Ignite#active(false) is used outside the tests? What is so wrong to use new IgniteCluster(INACTIVE, force:true)? To the point, the PR is here: пт, 14 февр. 2020 г. в 15:57, Вячеслав Коптилин : > Hello Vladimir, > > > adding a new method with force flag means old methods change their > behavior: > I don't think that changing the behavior of public API is the right way. > Moreover, I agree with Alex that there is no need to introduce a > "confirmation" flag to the java API. > > Thanks, > S. > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > > > Alexey, adding a new method with force flag means old methods change > their > > behavior: they are considered as executed without ‘force‘ flag and can > fail > > to prevent data loss. Ignite and IgniteMXBean are different interfaces. > > Unfortunately, they have same method > > > > void active(boolean active) > > > > When executed as IgniteMXBean it should fail if user can lose data. When > > executed from code via interface Ignite probably not. To solve this I > > suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST and > > other API. > > > > пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > Igniters, > > > > > > Do we really need the confirmation flag on the public API? I absolutely > > > agree on the CLI and MXBean, but what is the reason for the flag in the > > > API? It will be specified at the compile time anyway and does not > prevent > > > any user error. > > > From the implementation point of view I see no contradiction - we can > add > > > the new method to the MXBean, but nothing forces us to add it to Ignite > > > interface - those interfaces are not related. > > > > > >
Re: JAVADOC fails on local build. Should it be checked on TC?
I merged changes for [1] to master. Next steps will be updating TC javadoc to fail on a string "Javadoc Warnings" [1] https://issues.apache.org/jira/browse/IGNITE-12644 Best regards, Ivan Pavlukhin сб, 8 февр. 2020 г. в 21:03, Ivan Pavlukhin : > > I created a ticket [1] and PR [2]. Please review. > > [1] https://issues.apache.org/jira/browse/IGNITE-12644 > [2] https://github.com/apache/ignite/pull/7356 > > Best regards, > Ivan Pavlukhin > > вт, 4 февр. 2020 г. в 17:51, Ivan Pavlukhin : > > > > The next bit I found out is that links like > > "https://hadoop.apache.org/docs/current/api/" > > are needed to have working links to external classes in javadocs (e.g. > > Hadoop ones). Unfortunately, we are not consistent with external > > links. E.g. references to JCache classes are rendered as plain text in > > our javadocs [1]. > > > > For now I suggest to drop Hadoop links from maven plugin configuration > > as well. If there are no objections I will prepare a PR. > > > > [1] > > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheEntryProcessor.html > > > > вт, 4 февр. 2020 г. в 15:04, Ivan Pavlukhin : > > > > > > Alexey, > > > > > > Thank you for the hint. Actually my bet was the same. But currently I > > > have no understanding how > > > "https://hadoop.apache.org/docs/current/api/" tag is used > > > and is it used at all. > > > > > > вт, 4 февр. 2020 г. в 15:01, Alexey Goncharuk > > > : > > > > > > > > Ivan, > > > > > > > > My guess would be that the link was included for the Hadoop Accelerator > > > > which we all agreed to remove [1]. I think it is safe to remove it from > > > > pom.xml completely. > > > > > > > > [1] > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Vyacheslav. What is your suggestion for the API? Single implementation for both Ignite#active(boolean) and IgniteMXBean#active(boolean) Should public java API continue to silently clears in-memory caches? > 14 февр. 2020 г., в 15:56, Вячеслав Коптилин > написал(а): > > Hello Vladimir, > >> adding a new method with force flag means old methods change their > behavior: > I don't think that changing the behavior of public API is the right way. > Moreover, I agree with Alex that there is no need to introduce a > "confirmation" flag to the java API. > > Thanks, > S. > > пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > >> Alexey, adding a new method with force flag means old methods change their >> behavior: they are considered as executed without ‘force‘ flag and can fail >> to prevent data loss. Ignite and IgniteMXBean are different interfaces. >> Unfortunately, they have same method >> >> void active(boolean active) >> >> When executed as IgniteMXBean it should fail if user can lose data. When >> executed from code via interface Ignite probably not. To solve this I >> suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST and >> other API. >> >> пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk >> : >> >>> Igniters, >>> >>> Do we really need the confirmation flag on the public API? I absolutely >>> agree on the CLI and MXBean, but what is the reason for the flag in the >>> API? It will be specified at the compile time anyway and does not prevent >>> any user error. >>> From the implementation point of view I see no contradiction - we can add >>> the new method to the MXBean, but nothing forces us to add it to Ignite >>> interface - those interfaces are not related. >>> >>
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hello Vladimir, > adding a new method with force flag means old methods change their behavior: I don't think that changing the behavior of public API is the right way. Moreover, I agree with Alex that there is no need to introduce a "confirmation" flag to the java API. Thanks, S. пт, 14 февр. 2020 г. в 15:38, Vladimir Steshin : > Alexey, adding a new method with force flag means old methods change their > behavior: they are considered as executed without ‘force‘ flag and can fail > to prevent data loss. Ignite and IgniteMXBean are different interfaces. > Unfortunately, they have same method > > void active(boolean active) > > When executed as IgniteMXBean it should fail if user can lose data. When > executed from code via interface Ignite probably not. To solve this I > suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST and > other API. > > пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk >: > > > Igniters, > > > > Do we really need the confirmation flag on the public API? I absolutely > > agree on the CLI and MXBean, but what is the reason for the flag in the > > API? It will be specified at the compile time anyway and does not prevent > > any user error. > > From the implementation point of view I see no contradiction - we can add > > the new method to the MXBean, but nothing forces us to add it to Ignite > > interface - those interfaces are not related. > > >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Alexey, adding a new method with force flag means old methods change their behavior: they are considered as executed without ‘force‘ flag and can fail to prevent data loss. Ignite and IgniteMXBean are different interfaces. Unfortunately, they have same method void active(boolean active) When executed as IgniteMXBean it should fail if user can lose data. When executed from code via interface Ignite probably not. To solve this I suggest to add ‘force’ flag for every deactivation mode: CLI/JMX/REST and other API. пт, 14 февр. 2020 г. в 15:20, Alexey Goncharuk : > Igniters, > > Do we really need the confirmation flag on the public API? I absolutely > agree on the CLI and MXBean, but what is the reason for the flag in the > API? It will be specified at the compile time anyway and does not prevent > any user error. > From the implementation point of view I see no contradiction - we can add > the new method to the MXBean, but nothing forces us to add it to Ignite > interface - those interfaces are not related. >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Alexey. The issue with the existing method. Single implementation for both Ignite#active(boolean) and IgniteMXBean#active(boolean) Should it continue to silently clears in-memory caches? > 14 февр. 2020 г., в 15:20, Alexey Goncharuk > написал(а): > > Igniters, > > Do we really need the confirmation flag on the public API? I absolutely > agree on the CLI and MXBean, but what is the reason for the flag in the > API? It will be specified at the compile time anyway and does not prevent > any user error. > From the implementation point of view I see no contradiction - we can add > the new method to the MXBean, but nothing forces us to add it to Ignite > interface - those interfaces are not related.
[jira] [Created] (IGNITE-12682) Remove IgniteMessageFactoryImpl.registerCustom() method
Andrey N. Gura created IGNITE-12682:
---
Summary: Remove IgniteMessageFactoryImpl.registerCustom() method
Key: IGNITE-12682
URL: https://issues.apache.org/jira/browse/IGNITE-12682
Project: Ignite
Issue Type: Improvement
Reporter: Andrey N. Gura
Assignee: Andrey N. Gura
Fix For: 2.9
{{IgniteMessageFactoryImpl.registerCustom()}} method should be removed because
it potentially dangerous and could lead to the problems at runtime. All tests
that use this method should be rewritten using plugins with own message
factories.
Also boilerplate code of plugins should be rewritten with
{{AbstractTestPluginProvider}} (see code review comment
https://github.com/apache/ignite/pull/7417#discussion_r379366291)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Igniters, Do we really need the confirmation flag on the public API? I absolutely agree on the CLI and MXBean, but what is the reason for the flag in the API? It will be specified at the compile time anyway and does not prevent any user error. >From the implementation point of view I see no contradiction - we can add the new method to the MXBean, but nothing forces us to add it to Ignite interface - those interfaces are not related.
[jira] [Created] (IGNITE-12681) IgniteShutdownOnSupplyMessageFailureTest
Alexey Goncharuk created IGNITE-12681: - Summary: IgniteShutdownOnSupplyMessageFailureTest Key: IGNITE-12681 URL: https://issues.apache.org/jira/browse/IGNITE-12681 Project: Ignite Issue Type: Test Reporter: Alexey Goncharuk The test checks that a node will be shut down by a failure handler by listening for NODE_LEFT event. However, if the node shutdown happens before a new node joins the cluster, the joining node will form a cluster by itself with topology version = 1 and no event will be fired. The test should be changed to specifically listen for the failure handler invocation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
IGNITE-10698 Task
Hello everyone, I've fixed codestyle issues in my branch and got "No possible blockers" in TC bot, so PR is currently ready for review. PR github link: https://github.com/apache/ignite/pull/7394 Jira task link: https://issues.apache.org/jira/browse/IGNITE-10698 TeamcityBot link: https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/7394/head&action=Latest
[jira] [Created] (IGNITE-12680) Cleanup unneeded explicit usages of SystemPropertiesRule
Kirill Tkalenko created IGNITE-12680:
Summary: Cleanup unneeded explicit usages of SystemPropertiesRule
Key: IGNITE-12680
URL: https://issues.apache.org/jira/browse/IGNITE-12680
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.9
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
In some test classes there are explicit declarations like:
{code}
@ClassRule public static final TestRule classRule = new SystemPropertiesRule();
{code}
Usually there is no need for it as class and instance rules needed for
\{{@WithSystemProperty}} annotation are already present in
\{{GridAbstractTest}}.
Let's cleanup such usages from code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hi, guys. I suggest mark as deprecated the IgniteMXBean#active(boolean) method. The Ignite#active(boolean), IgniteMXBean#active(), IgniteCluster#active(boolean) methods already deprecated and told to use the #state(ClusterState)} instead. Should the old IgniteMXBean#state(ClusterState) method throw a not-safe deactivation exception? Or it should just be marked as deprecated? пт, 14 февр. 2020 г. в 12:18, Nikolay Izhikov : > > Hello, Vladimir. > > I think we should do the following: > > * Update Ignite documentation and write down the fact that in-memory cache > cleared on deactivation. > * Disallow, by default, deactivation of the cluster that has in-memory cache > with proper error message > «Your cluster has in-memory cache configured. During deactivation all > data from these caches will be cleared!» > * Add «—force» flag to deactivate command so administrator can force > deactivation for cluster that has both - persistent and in-memory caches > configured. > > Changes in the source code should include: > > 1. JMX beans - IgniteMXBean. > 2. Java API - Ignite#active and Ignite#cluster#state - because active method > have the same implementation as `IgniteMBBean#active` > 3. CLI utility - control.sh > > Igniters, what do you think? > > > 13 февр. 2020 г., в 14:19, Vladimir Steshin написал(а): > > > > Hello, everyone. > > > > > > > > I’d like to propose to make behavior of cluster deactivation the same > > independently on usage type: in code or manually with control.sh or JMX. > > Let’s put it in one place like IgniteCluster#state(ClusterState newState, > > boolean force) and require forced call for in-memory data. > > > > While I was doing the ticket I realized that the suggested previously > > solution cannot be complete. To prevent data loss via JMX I would need to > > stop executing IgniteMXBean#active(false). But this causes stopping of > > Ignite#active(false) too. The problem relies in single implementation for > > both interfaces in IgniteKernal#active(boolean). It has to act differently > > depending on where it is called from: JMX/CLI or code. > > > > > > > >Any thoughts? > > > > чт, 30 янв. 2020 г. в 13:08, Alexey Goncharuk : > > > >> I agree on CLI and JMX because those interfaces can be used by a system > >> administrator and can be invoked by mistake. > >> > >> As for the Java API, personally, I find it strange to add 'force' or > >> 'confirm' flags to it because it is very unlikely that such an invocation > >> is done by mistake. Such mistakes are caught during the testing phase and > >> developers will end up hard-coding 'true' as a flag anyways. > >> > -- Best wishes, Amelchev Nikita
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
Hello, Vladimir. I think we should do the following: * Update Ignite documentation and write down the fact that in-memory cache cleared on deactivation. * Disallow, by default, deactivation of the cluster that has in-memory cache with proper error message «Your cluster has in-memory cache configured. During deactivation all data from these caches will be cleared!» * Add «—force» flag to deactivate command so administrator can force deactivation for cluster that has both - persistent and in-memory caches configured. Changes in the source code should include: 1. JMX beans - IgniteMXBean. 2. Java API - Ignite#active and Ignite#cluster#state - because active method have the same implementation as `IgniteMBBean#active` 3. CLI utility - control.sh Igniters, what do you think? > 13 февр. 2020 г., в 14:19, Vladimir Steshin написал(а): > > Hello, everyone. > > > > I’d like to propose to make behavior of cluster deactivation the same > independently on usage type: in code or manually with control.sh or JMX. > Let’s put it in one place like IgniteCluster#state(ClusterState newState, > boolean force) and require forced call for in-memory data. > > While I was doing the ticket I realized that the suggested previously > solution cannot be complete. To prevent data loss via JMX I would need to > stop executing IgniteMXBean#active(false). But this causes stopping of > Ignite#active(false) too. The problem relies in single implementation for > both interfaces in IgniteKernal#active(boolean). It has to act differently > depending on where it is called from: JMX/CLI or code. > > > >Any thoughts? > > чт, 30 янв. 2020 г. в 13:08, Alexey Goncharuk : > >> I agree on CLI and JMX because those interfaces can be used by a system >> administrator and can be invoked by mistake. >> >> As for the Java API, personally, I find it strange to add 'force' or >> 'confirm' flags to it because it is very unlikely that such an invocation >> is done by mistake. Such mistakes are caught during the testing phase and >> developers will end up hard-coding 'true' as a flag anyways. >>
