[jira] [Commented] (IGNITE-13845) Add checkpoint lock hold metrics

2020-12-14 Thread Ignite TC Bot (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249519#comment-17249519
 ] 

Ignite TC Bot commented on IGNITE-13845:


{panel:title=Branch: [pull/8572/head] Base: [master] : No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/8572/head] Base: [master] : New Tests 
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 2{color} [[tests 
1|https://ci.ignite.apache.org/viewLog.html?buildId=5790030]]
* {color:#013220}IgnitePdsTestSuite2: 
IgniteDataStorageMetricsSelfTest.testCheckpointMetrics - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5790058buildTypeId=IgniteTests24Java8_RunAll]

> Add checkpoint lock hold metrics
> 
>
> Key: IGNITE-13845
> URL: https://issues.apache.org/jira/browse/IGNITE-13845
> Project: Ignite
>  Issue Type: New Feature
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add last checkpoint lock hold duration and durations histogram metrics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13854) Add documentation for the cluster performance profiling tool

2020-12-14 Thread Amelchev Nikita (Jira)
Amelchev Nikita created IGNITE-13854:


 Summary: Add documentation for the cluster performance profiling 
tool
 Key: IGNITE-13854
 URL: https://issues.apache.org/jira/browse/IGNITE-13854
 Project: Ignite
  Issue Type: Sub-task
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita


Add documentation for the cluster performance profiling tool.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (IGNITE-13451) Provide the ability to set up default performance statistics properties

2020-12-14 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita resolved IGNITE-13451.
--
Resolution: Done

> Provide the ability to set up default performance statistics properties
> ---
>
> Key: IGNITE-13451
> URL: https://issues.apache.org/jira/browse/IGNITE-13451
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>
> Provide the ability to set up default performance statistics properties:
> - DFLT_FILE_MAX_SIZE
> - DFLT_BUFFER_SIZE
> - DFLT_FLUSH_SIZE
> - DFLT_MAX_CACHED_STRINGS_COUNT



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (IGNITE-13252) Provide the ability to cache strings

2020-12-14 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita resolved IGNITE-13252.
--
Resolution: Done

> Provide the ability to cache strings
> 
>
> Key: IGNITE-13252
> URL: https://issues.apache.org/jira/browse/IGNITE-13252
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>
> Strings that performance statistics writes can be cached with id. It will 
> reduce affect on performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Resolved] (IGNITE-13452) Create a new performance statistics file instead of deleting the existing

2020-12-14 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita resolved IGNITE-13452.
--
Resolution: Done

> Create a new performance statistics file instead of deleting the existing
> -
>
> Key: IGNITE-13452
> URL: https://issues.apache.org/jira/browse/IGNITE-13452
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>
> Create a new performance statistics file instead of deleting the existing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13847) Make GridEncryptionManager#onWalSegmentRemoved async

2020-12-14 Thread Kirill Tkalenko (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249490#comment-17249490
 ] 

Kirill Tkalenko commented on IGNITE-13847:
--

Used the [PR's|https://github.com/xtern/ignite/pull/44/files] from the 
IGNITE-13831 from [~xtern]. 

> Make GridEncryptionManager#onWalSegmentRemoved async
> 
>
> Key: IGNITE-13847
> URL: https://issues.apache.org/jira/browse/IGNITE-13847
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
>  Labels: IEP-18
> Fix For: 2.10
>
>
> When implementing IGNITE-13831 I was faced with deadlock.
> When execute *FileWriteAheadLogManager#rollOver*, begin to clean WAL archive 
> since we have reached the *DataStorageConfiguration#maxWalArchiveSize*, after 
> deleting a segment, execute the *GridEncryptionManager#onWalSegmentRemoved* 
> that wants to write to the metastore, but it will not succeed, since it will 
> wait for *FileWriteAheadLogManager#rollOver*.
> I suggest making the *GridEncryptionManager#onWalSegmentRemoved* asynchronous 
> in a separate pool, for example, as a *CacheGroupPageScanner#singleExecSvc*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (IGNITE-13847) Make GridEncryptionManager#onWalSegmentRemoved async

2020-12-14 Thread Kirill Tkalenko (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko reassigned IGNITE-13847:


Assignee: Kirill Tkalenko

> Make GridEncryptionManager#onWalSegmentRemoved async
> 
>
> Key: IGNITE-13847
> URL: https://issues.apache.org/jira/browse/IGNITE-13847
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
>  Labels: IEP-18
> Fix For: 2.10
>
>
> When implementing IGNITE-13831 I was faced with deadlock.
> When execute *FileWriteAheadLogManager#rollOver*, begin to clean WAL archive 
> since we have reached the *DataStorageConfiguration#maxWalArchiveSize*, after 
> deleting a segment, execute the *GridEncryptionManager#onWalSegmentRemoved* 
> that wants to write to the metastore, but it will not succeed, since it will 
> wait for *FileWriteAheadLogManager#rollOver*.
> I suggest making the *GridEncryptionManager#onWalSegmentRemoved* asynchronous 
> in a separate pool, for example, as a *CacheGroupPageScanner#singleExecSvc*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13853) Add APIs, examples, language selector to the docs menu

2020-12-14 Thread Denis A. Magda (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis A. Magda updated IGNITE-13853:

Description: 
There is user feedback that suggests we need to rework our navigation menus of 
the website and docs to improve the discoverability of APIs and examples. Let's 
do the following.

 

*Ignite Website Top-Level Navigation Menu 
([https://ignite.apache.org)|https://ignite.apache.org)/]*
 * Add the "Docs" menu item before the "Features" item. The "Docs" _is a single 
link_ (not a dropdown menu) that directs to the root of the latest technical 
documentation: [https://ignite.apache.org/docs/latest/]
 * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
(Chinese)" and "APIs" from this menu section.
 * Remove "Events" from the top-level menu. Add the items/links from the menu 
to the "Community"

 

*Technical Documentation Top-Level Navigation Menu 
([https://ignite.apache.org/docs/latest/])*
 * Remove the Github icon from the menu
 * Add "APIs" after the "version" selector. The menu lists "Java", "C#/.NET", 
"C++", "Scala". The "version" selector defines what version of an API is to be 
opened. For instance, if a user chooses version "2.9.1" (to be released soon) 
then "Java" of that version is to be opened. If instead, the user wants to see 
the docs and APIs for "2.9.0", he just needs to selects that version from the 
version menu. 
 * Add "Examples" after the "APIs". Add the following menu items:
 ** "Java": [https://github.com/apache/ignite/tree/master/examples]
 ** "C#/.NET": 
[https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
 ** "C++": 
[https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
 ** "Python": 
[https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
 ** "Node.JS": 
[https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
 ** "PHP": 
[https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
 * Add a language selector drop-down list after the "search" text box. See how 
it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] 
We'll have just two menu items there:
 ** "English": points out to the root of the main docs 
[https://ignite.apache.org/docs/latest/]
 ** "Chinese": [https://www.ignite-service.cn/doc/java/]

  was:
There is user feedback that suggests we need to rework our navigation menus of 
the website and docs to improve the discoverability of APIs and examples. Let's 
do the following.

 

*Ignite Website Top-Level Navigation Menu 
([https://ignite.apache.org)|https://ignite.apache.org)/]*
 * Add the "Docs" menu item before the "Features" item. The "Docs" _is a single 
link_ (not a dropdown menu) that directs to the root of the latest technical 
documentation: [https://ignite.apache.org/docs/latest/]
 * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
(Chinese)" and "APIs" from this menu section.
 * Remove "Events" from the top-level menu. Add the items/links from the menu 
to the "Community"

 

*Technical Documentation Top-Level Navigation Menu 
([https://ignite.apache.org/docs/latest/])*
 * Remove the Github icon from the menu
 * Add "APIs" after the "version" selector. The menu lists "JavaDocs", 
"C#/.NET", "C++", "Scala". The "version" selector defines what version of an 
API is to be opened. For instance, if a user chooses version "2.9.1" (to be 
released soon) then "JavaDocs" of that version is to be opened. If instead, the 
user wants to see the docs and APIs for "2.9.0", he just needs to selects that 
version from the version menu. 
 * Add "Examples" after the "APIs". Add the following menu items:
 ** "Java": [https://github.com/apache/ignite/tree/master/examples]
 ** "C#/.NET": 
[https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
 ** "C++": 
[https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
 ** "Python": 
[https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
 ** "Node.JS": 
[https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
 ** "PHP": 
[https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
 * Add a language selector drop-down list after the "search" text box. See how 
it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] 
We'll have just two menu items there:
 ** "English": points out to the root of the main docs 
[https://ignite.apache.org/docs/latest/]
 ** "Chinese": [https://www.ignite-service.cn/doc/java/]


> Add APIs, examples, language selector to the docs menu
> --
>
> Key: IGNITE-13853
> URL: https://issues.apache.org/jira/browse/IGNITE-13853
> Project: Ignite
>  Issue Type: Improvement
>  

[jira] [Updated] (IGNITE-13853) Add APIs, examples, language selector to the docs menu

2020-12-14 Thread Denis A. Magda (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis A. Magda updated IGNITE-13853:

Description: 
There is user feedback that suggests we need to rework our navigation menus of 
the website and docs to improve the discoverability of APIs and examples. Let's 
do the following.

 

*Ignite Website Top-Level Navigation Menu 
([https://ignite.apache.org)|https://ignite.apache.org)/]*
 * Add the "Docs" menu item before the "Features" item. The "Docs" _is a single 
link_ (not a dropdown menu) that directs to the root of the latest technical 
documentation: [https://ignite.apache.org/docs/latest/]
 * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
(Chinese)" and "APIs" from this menu section.
 * Remove "Events" from the top-level menu. Add the items/links from the menu 
to the "Community"

 

*Technical Documentation Top-Level Navigation Menu 
([https://ignite.apache.org/docs/latest/])*
 * Remove the Github icon from the menu
 * Add "APIs" after the "version" selector. The menu lists "JavaDocs", 
"C#/.NET", "C++", "Scala". The "version" selector defines what version of an 
API is to be opened. For instance, if a user chooses version "2.9.1" (to be 
released soon) then "JavaDocs" of that version is to be opened. If instead, the 
user wants to see the docs and APIs for "2.9.0", he just needs to selects that 
version from the version menu. 
 * Add "Examples" after the "APIs". Add the following menu items:
 ** "Java": [https://github.com/apache/ignite/tree/master/examples]
 ** "C#/.NET": 
[https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
 ** "C++": 
[https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
 ** "Python": 
[https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
 ** "Node.JS": 
[https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
 ** "PHP": 
[https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
 * Add a language selector drop-down list after the "search" text box. See how 
it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] 
We'll have just two menu items there:
 ** "English": points out to the root of the main docs 
[https://ignite.apache.org/docs/latest/]
 ** "Chinese": [https://www.ignite-service.cn/doc/java/]

  was:
There is user feedback that suggests we need to rework our navigation menus of 
the website and docs to improve the discoverability of APIs and examples. Let's 
do the following.

 

*Ignite Website Top-Level Navigation Menu 
([https://ignite.apache.org)|https://ignite.apache.org)/]*
 * Add the "Docs" menu item to the menu placing it before the "Features" item. 
The "Docs" _is a single link_ (no dropdown menu) that directs to the root of 
the latest technical documentation: [https://ignite.apache.org/docs/latest/]
 * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
(Chinese)" and "APIs" from this menu section.
 * Remove "Events" from the top-level menu. Add the items/links from the menu 
to the "Community"

 

*Technical Documentation Top-Level Navigation Menu 
([https://ignite.apache.org/docs/latest/])*
 * Remove the Github icon from the menu
 * Add "APIs" after the "version" selector. The menu lists "JavaDocs", 
"C#/.NET", "C++", "Scala". The "version" selector defines what version of an 
API is to be opened. For instance, if a user chooses version "2.9.1" (to be 
released soon) then "JavaDocs" of that version is to be opened. If instead, the 
user wants to see the docs and APIs for "2.9.0", he just needs to selects that 
version from the version menu. 
 * Add "Examples" after the "APIs". Add the following menu items:
 ** "Java": [https://github.com/apache/ignite/tree/master/examples]
 ** "C#/.NET": 
[https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
 ** "C++": 
[https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
 ** "Python": 
[https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
 ** "Node.JS": 
[https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
 ** "PHP": 
[https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
 * Add a language selector drop-down list after the "search" text box. See how 
it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] 
We'll have just two menu items there:
 ** "English": points out to the root of the main docs 
[https://ignite.apache.org/docs/latest/]
 ** "Chinese": https://www.ignite-service.cn/doc/java/


> Add APIs, examples, language selector to the docs menu
> --
>
> Key: IGNITE-13853
> URL: https://issues.apache.org/jira/browse/IGNITE-13853
> Project: Ignite
>  

[jira] [Created] (IGNITE-13853) Add APIs, examples, language selector to the docs menu

2020-12-14 Thread Denis A. Magda (Jira)
Denis A. Magda created IGNITE-13853:
---

 Summary: Add APIs, examples, language selector to the docs menu
 Key: IGNITE-13853
 URL: https://issues.apache.org/jira/browse/IGNITE-13853
 Project: Ignite
  Issue Type: Improvement
  Components: documentation, website
Reporter: Denis A. Magda
 Fix For: 2.9.1


There is user feedback that suggests we need to rework our navigation menus of 
the website and docs to improve the discoverability of APIs and examples. Let's 
do the following.

 

*Ignite Website Top-Level Navigation Menu 
([https://ignite.apache.org)|https://ignite.apache.org)/]*
 * Add the "Docs" menu item to the menu placing it before the "Features" item. 
The "Docs" _is a single link_ (no dropdown menu) that directs to the root of 
the latest technical documentation: [https://ignite.apache.org/docs/latest/]
 * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
(Chinese)" and "APIs" from this menu section.
 * Remove "Events" from the top-level menu. Add the items/links from the menu 
to the "Community"

 

*Technical Documentation Top-Level Navigation Menu 
([https://ignite.apache.org/docs/latest/])*
 * Remove the Github icon from the menu
 * Add "APIs" after the "version" selector. The menu lists "JavaDocs", 
"C#/.NET", "C++", "Scala". The "version" selector defines what version of an 
API is to be opened. For instance, if a user chooses version "2.9.1" (to be 
released soon) then "JavaDocs" of that version is to be opened. If instead, the 
user wants to see the docs and APIs for "2.9.0", he just needs to selects that 
version from the version menu. 
 * Add "Examples" after the "APIs". Add the following menu items:
 ** "Java": [https://github.com/apache/ignite/tree/master/examples]
 ** "C#/.NET": 
[https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
 ** "C++": 
[https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
 ** "Python": 
[https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
 ** "Node.JS": 
[https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
 ** "PHP": 
[https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
 * Add a language selector drop-down list after the "search" text box. See how 
it's implemented here: [https://www.confluent.io.|https://www.confluent.io./] 
We'll have just two menu items there:
 ** "English": points out to the root of the main docs 
[https://ignite.apache.org/docs/latest/]
 ** "Chinese": https://www.ignite-service.cn/doc/java/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13853) Add APIs, examples, language selector to the docs menu

2020-12-14 Thread Denis A. Magda (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249466#comment-17249466
 ] 

Denis A. Magda commented on IGNITE-13853:
-

[~mstekl], could you please help with this task?

> Add APIs, examples, language selector to the docs menu
> --
>
> Key: IGNITE-13853
> URL: https://issues.apache.org/jira/browse/IGNITE-13853
> Project: Ignite
>  Issue Type: Improvement
>  Components: documentation, website
>Reporter: Denis A. Magda
>Priority: Major
> Fix For: 2.9.1
>
>
> There is user feedback that suggests we need to rework our navigation menus 
> of the website and docs to improve the discoverability of APIs and examples. 
> Let's do the following.
>  
> *Ignite Website Top-Level Navigation Menu 
> ([https://ignite.apache.org)|https://ignite.apache.org)/]*
>  * Add the "Docs" menu item to the menu placing it before the "Features" 
> item. The "Docs" _is a single link_ (no dropdown menu) that directs to the 
> root of the latest technical documentation: 
> [https://ignite.apache.org/docs/latest/]
>  * Rename "Resources" to "Learn More". Remove "Documentation", "Documentation 
> (Chinese)" and "APIs" from this menu section.
>  * Remove "Events" from the top-level menu. Add the items/links from the menu 
> to the "Community"
>  
> *Technical Documentation Top-Level Navigation Menu 
> ([https://ignite.apache.org/docs/latest/])*
>  * Remove the Github icon from the menu
>  * Add "APIs" after the "version" selector. The menu lists "JavaDocs", 
> "C#/.NET", "C++", "Scala". The "version" selector defines what version of an 
> API is to be opened. For instance, if a user chooses version "2.9.1" (to be 
> released soon) then "JavaDocs" of that version is to be opened. If instead, 
> the user wants to see the docs and APIs for "2.9.0", he just needs to selects 
> that version from the version menu. 
>  * Add "Examples" after the "APIs". Add the following menu items:
>  ** "Java": [https://github.com/apache/ignite/tree/master/examples]
>  ** "C#/.NET": 
> [https://github.com/apache/ignite/tree/master/modules/platforms/dotnet/examples]
>  ** "C++": 
> [https://github.com/apache/ignite/tree/master/modules/platforms/cpp/examples]
>  ** "Python": 
> [https://github.com/apache/ignite/tree/master/modules/platforms/python/examples]
>  ** "Node.JS": 
> [https://github.com/apache/ignite/tree/master/modules/platforms/nodejs/examples]
>  ** "PHP": 
> [https://github.com/apache/ignite/tree/master/modules/platforms/php/examples]
>  * Add a language selector drop-down list after the "search" text box. See 
> how it's implemented here: 
> [https://www.confluent.io.|https://www.confluent.io./] We'll have just two 
> menu items there:
>  ** "English": points out to the root of the main docs 
> [https://ignite.apache.org/docs/latest/]
>  ** "Chinese": https://www.ignite-service.cn/doc/java/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13852) Ascii tables for new unified cli tool

2020-12-14 Thread Kirill Gusakov (Jira)
Kirill Gusakov created IGNITE-13852:
---

 Summary: Ascii tables for new unified cli tool
 Key: IGNITE-13852
 URL: https://issues.apache.org/jira/browse/IGNITE-13852
 Project: Ignite
  Issue Type: Task
Reporter: Kirill Gusakov


Implement Ascii tables with support of Ansi symbols and custom borders



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (IGNITE-12824) Interoperable Ignite.NET Dates

2020-12-14 Thread Nikolay Izhikov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249138#comment-17249138
 ] 

Nikolay Izhikov edited comment on IGNITE-12824 at 12/14/20, 5:18 PM:
-

Hello, [~kukushal], [~ptupitsyn]

I investigated this issue and find a bug, but not one described :)

1. For now, We can't use {{DateTime}} type as a key or value for a cache. I 
fixed it in the [PR #8568|https://github.com/apache/ignite/pull/8568].

2. I just delete explicit exception for {{DateTimeKind.Local}} dates and found 
no issue. 
Please, [take a look at my 
test|https://github.com/apache/ignite/pull/8568/files#diff-978bbc38ff93b1f1a62517f532de7bf4d0a320de32550515a062abb830c58164R959]
Can you, please, clarify the circumstances to reproduce the issue from the 
description.

For now, it looks like we just can remove a restriction for non UTC dates and 
that's all :)


was (Author: nizhikov):
Hello, [~kukushal], [~ptupitsyn]

I investigated this issue and find a bug, but not one described in the 
description :)

1. For now, We can't use {{DateTime}} type as a key or value for a cache. I 
fixed it in the [PR #8568|https://github.com/apache/ignite/pull/8568].

2. I just delete explicit exception for {{DateTimeKind.Local}} dates and found 
no issue. 
Please, [take a look at my 
test|https://github.com/apache/ignite/pull/8568/files#diff-978bbc38ff93b1f1a62517f532de7bf4d0a320de32550515a062abb830c58164R959]
Can you, please, clarify the circumstances to reproduce the issue from the 
description.

For now, it looks like we just can remove a restriction for non UTC dates and 
that's all :)

> Interoperable Ignite.NET Dates
> --
>
> Key: IGNITE-12824
> URL: https://issues.apache.org/jira/browse/IGNITE-12824
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Affects Versions: 2.8
>Reporter: Alexey Kukushkin
>Assignee: Nikolay Izhikov
>Priority: Major
>  Labels: .NET, iep-54, ignite-3, sbcf
> Fix For: 3.0
>
> Attachments: IGNITE-12824-from-2.9.0.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> *+The Problem+* 
>  Presently .NET API writes dates as composite Ignite objects. Only .NET 
> clients can read such dates: any other client (JDBC, Java, etc) does not 
> understand it without custom deserialization.
>   
>  It is still possible to configure .NET serialization to write dates as 
> Ignite dates - see [DateTime Serialization 
> note|https://ignite.apache.org/docs/latest/net-specific/net-platform-interoperability#types-compatibility].
>  But then Ignite accepts only UTC dates, requiring the application developers 
> to convert local dates to UTC dates and back. This task is not trivial: 
> [DateTime.ToUniversalTime|https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netcore-3.1]
>  uses calendars different from Java (and the .NET calendars are the invalid 
> ones - especially for pre-1990 dates).
>   
>  The motivation for the current default behavior was probably the desire to 
> keep the Time Zone information: Ignite dates do not store time zones.
>   
>  In our experience interoperability is more important than storing time zone 
> info.
>   
>  *+The Solution+*
>  # Always write .NET dates as portable Ignite dates: get rid of the 
> {{BinaryReflectiveSerializer.ForceTimestamp}} flag that currently triggers 
> this behavior.
>  Keep the {{ForceTimestamp}} flag if saving .NET dates as transparent objects 
> seems a useful case.
>  # Automatically convert Local dates to UTC and back *inside* Ignite.NET. 
>  Add a {{BinaryReflectiveSerializer.UtcDate flag}} to disable this 
> conversion. This prevents loosing the {{DateTime.Kind}} property of UTC 
> dates.  {{UtcDate}} set to true implies the existing behavior: Ignite gets 
> UTC dates and throws a "Date must be in UTC" exception on an attempt to write 
> a Local date. The {{UtcDate}} flag is false by default.
>  # Use [NodaTime|https://nodatime.org/] for UTC<->Local conversions. Noda 
> time uses Java calendars making the conversion truly portable.
>  
>  *+The Benefits+*
>  # Portable dates is a more frequent use-case than storing time zone info for 
> every date in Ignite. This becomes default behavior and the developers do not 
> need to always explicitly configure it.
>  # Non-trivial code to make the truly portable UTC<->Local conversion is 
> implemented once inside Ignite instead of having every Ignite.NET application 
> implementing it.
> +*References*+
>  * [Dev-List 
> Discussion|http://apache-ignite-developers.2346864.n4.nabble.com/Interoperable-Ignite-NET-Dates-td49946.html]
>  * IEP-TBD



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-12824) Interoperable Ignite.NET Dates

2020-12-14 Thread Nikolay Izhikov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249138#comment-17249138
 ] 

Nikolay Izhikov commented on IGNITE-12824:
--

Hello, [~kukushal], [~ptupitsyn]

I investigated this issue and find a bug, but not one described in the 
description :)

1. For now, We can't use {{DateTime}} type as a key or value for a cache. I 
fixed it in the [PR #8568|https://github.com/apache/ignite/pull/8568].

2. I just delete explicit exception for {{DateTimeKind.Local}} dates and found 
no issue. 
Please, [take a look at my 
test|https://github.com/apache/ignite/pull/8568/files#diff-978bbc38ff93b1f1a62517f532de7bf4d0a320de32550515a062abb830c58164R959]
Can you, please, clarify the circumstances to reproduce the issue from the 
description.

For now, it looks like we just can remove a restriction for non UTC dates and 
that's all :)

> Interoperable Ignite.NET Dates
> --
>
> Key: IGNITE-12824
> URL: https://issues.apache.org/jira/browse/IGNITE-12824
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Affects Versions: 2.8
>Reporter: Alexey Kukushkin
>Assignee: Nikolay Izhikov
>Priority: Major
>  Labels: .NET, iep-54, ignite-3, sbcf
> Fix For: 3.0
>
> Attachments: IGNITE-12824-from-2.9.0.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> *+The Problem+* 
>  Presently .NET API writes dates as composite Ignite objects. Only .NET 
> clients can read such dates: any other client (JDBC, Java, etc) does not 
> understand it without custom deserialization.
>   
>  It is still possible to configure .NET serialization to write dates as 
> Ignite dates - see [DateTime Serialization 
> note|https://ignite.apache.org/docs/latest/net-specific/net-platform-interoperability#types-compatibility].
>  But then Ignite accepts only UTC dates, requiring the application developers 
> to convert local dates to UTC dates and back. This task is not trivial: 
> [DateTime.ToUniversalTime|https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netcore-3.1]
>  uses calendars different from Java (and the .NET calendars are the invalid 
> ones - especially for pre-1990 dates).
>   
>  The motivation for the current default behavior was probably the desire to 
> keep the Time Zone information: Ignite dates do not store time zones.
>   
>  In our experience interoperability is more important than storing time zone 
> info.
>   
>  *+The Solution+*
>  # Always write .NET dates as portable Ignite dates: get rid of the 
> {{BinaryReflectiveSerializer.ForceTimestamp}} flag that currently triggers 
> this behavior.
>  Keep the {{ForceTimestamp}} flag if saving .NET dates as transparent objects 
> seems a useful case.
>  # Automatically convert Local dates to UTC and back *inside* Ignite.NET. 
>  Add a {{BinaryReflectiveSerializer.UtcDate flag}} to disable this 
> conversion. This prevents loosing the {{DateTime.Kind}} property of UTC 
> dates.  {{UtcDate}} set to true implies the existing behavior: Ignite gets 
> UTC dates and throws a "Date must be in UTC" exception on an attempt to write 
> a Local date. The {{UtcDate}} flag is false by default.
>  # Use [NodaTime|https://nodatime.org/] for UTC<->Local conversions. Noda 
> time uses Java calendars making the conversion truly portable.
>  
>  *+The Benefits+*
>  # Portable dates is a more frequent use-case than storing time zone info for 
> every date in Ignite. This becomes default behavior and the developers do not 
> need to always explicitly configure it.
>  # Non-trivial code to make the truly portable UTC<->Local conversion is 
> implemented once inside Ignite instead of having every Ignite.NET application 
> implementing it.
> +*References*+
>  * [Dev-List 
> Discussion|http://apache-ignite-developers.2346864.n4.nabble.com/Interoperable-Ignite-NET-Dates-td49946.html]
>  * IEP-TBD



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-12892) Clarify WAL archive size configuration

2020-12-14 Thread shivakumar (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249085#comment-17249085
 ] 

shivakumar commented on IGNITE-12892:
-

Hi [~YAMolochkov]

Could you please consider this for 2.9.1 release ? It seems this code change 
(also IGNITE-13373 but not sure which one exactly fixed the issue) solves major 
blocker issue described here 

[http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html]

[http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html]

 

Regards

Shiva

> Clarify WAL archive size configuration
> --
>
> Key: IGNITE-12892
> URL: https://issues.apache.org/jira/browse/IGNITE-12892
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Actual maximum size of WAL archive that can be reserved for historical 
> rebalance is calculated as minimum of three properties:
>  # DataStorageConfiguration#walHistSize (units: number of checkpoints)
>  # DataStorageConfiguration#maxWalArchiveSize (units: bytes)
>  # IgniteSystemProperties#IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE 
> (units: number of checkpoints)
> The logic is a little unclear, so I propose following changes:
>  # Stop using walHistSize at all (it's already deprecated) for WAL truncation
>  # Use IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE only in case WAL archive 
> is managed externally (so we limit the quantity of checkpoints stored in 
> memory, but don't remove WAL files)
>  # Use -1 for maxWalArchiveSize (instead of Long.MAX_VALUE) to disable WAL 
> truncation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13373) WAL segmentns do not released on releaseHistoryForPreloading()

2020-12-14 Thread shivakumar (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249084#comment-17249084
 ] 

shivakumar commented on IGNITE-13373:
-

Hi [~YAMolochkov]

Could you please consider this for 2.9.1 release ? It seems this code change 
(also IGNITE-12892) solves major blocker issue described here 

[http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html]

[http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html]

 

Regards

Shiva

> WAL segmentns do not released on releaseHistoryForPreloading()
> --
>
> Key: IGNITE-13373
> URL: https://issues.apache.org/jira/browse/IGNITE-13373
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Alexander Lapin
>Priority: Major
>  Labels: 2.9.1-rc
> Fix For: 2.10
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> * Reserve/releaseHistoryForPreloading() was reworked, now we store oldest 
> WALPointer that was reserved during reserveHistoryForPreloading in 
> reservedForPreloading field. As a result it's possible to release wall 
> reservation on releaseHIstoryForPreloading().
>  * searchAndReserveCheckpoints() was slightly refactored: now it returns not 
> only an earliestValidCheckpoints but also an oldest reservedCheckpoint, so 
> that there’s no need to recalculate it within reserveHistoryForExchange().
>  *



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (IGNITE-13190) Core defragmentation functions

2020-12-14 Thread Maksim Timonin (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249069#comment-17249069
 ] 

Maksim Timonin edited comment on IGNITE-13190 at 12/14/20, 3:56 PM:


[~sergeychugunov] [~ibessonov] [~mmuzaf]

Hi! I'm was discovering how defragmentation work and found that defragmentation 
performed only for single index per table [1]. Is it correct behavior?

Also there is fixed offset 2 to get the index. It may lead to 
ArrayOutOfBoundsException in case of rebuilding hash index [2]. But I'm not 
sure whether this situation is possible or not.

[1] 
[IndexingDefragmentation.java#L124|https://github.com/apache/ignite/blob/dc4b71b230e289e5b0d99b00e0632c9d1d0e6e82/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/defragmentation/IndexingDefragmentation.java#L124]

[2] 
[GridH2Table.java#L1163|https://github.com/apache/ignite/blob/2fbbb676386515ea881e4e61f08864d6bc93225a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridH2Table.java#L1163]


was (Author: timonin.maksim):
[~sergeychugunov] [~ibessonov] [~mmuzaf]

Hi! I'm was discovering how defragmentation work and found that defragmentation 
performed only for single index per table [1]. Is it correct behavior?

Also there is fixed offset 2 to get the index. It may lead to 
ArrayOutOfBoundsException in case of rebuilding hash index [2]. I'm not sure 
whether this situation is possible or not.

[1] 
[IndexingDefragmentation.java#L124|https://github.com/apache/ignite/blob/dc4b71b230e289e5b0d99b00e0632c9d1d0e6e82/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/defragmentation/IndexingDefragmentation.java#L124]

[2] 
[GridH2Table.java#L1163|https://github.com/apache/ignite/blob/2fbbb676386515ea881e4e61f08864d6bc93225a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridH2Table.java#L1163]

> Core defragmentation functions
> --
>
> Key: IGNITE-13190
> URL: https://issues.apache.org/jira/browse/IGNITE-13190
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Sergey Chugunov
>Assignee: Ivan Bessonov
>Priority: Major
>  Labels: IEP-47
> Fix For: 2.10
>
>  Time Spent: 20h 50m
>  Remaining Estimate: 0h
>
> The following set of functions covering defragmentation happy-case needed:
>  * Initialization of defragmentation manager when node is started in 
> maintenance mode.
>  * Information about partition files is gathered by defrag mgr.
>  * For each partition file corresponding file of defragmented partition is 
> created and initialized.
>  * Keys are transferred from old partitions to new partitions.
>  * Checkpointer is aware of new partition files and flushes defragmented 
> memory to new partition files.
>  
> No fault-tolerance code nor index defragmentation mappings are needed in this 
> task.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13190) Core defragmentation functions

2020-12-14 Thread Maksim Timonin (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249069#comment-17249069
 ] 

Maksim Timonin commented on IGNITE-13190:
-

[~sergeychugunov] [~ibessonov] [~mmuzaf]

Hi! I'm was discovering how defragmentation work and found that defragmentation 
performed only for single index per table [1]. Is it correct behavior?

Also there is fixed offset 2 to get the index. It may lead to 
ArrayOutOfBoundsException in case of rebuilding hash index [2]. I'm not sure 
whether this situation is possible or not.

[1] 
[IndexingDefragmentation.java#L124|https://github.com/apache/ignite/blob/dc4b71b230e289e5b0d99b00e0632c9d1d0e6e82/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/defragmentation/IndexingDefragmentation.java#L124]

[2] 
[GridH2Table.java#L1163|https://github.com/apache/ignite/blob/2fbbb676386515ea881e4e61f08864d6bc93225a/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridH2Table.java#L1163]

> Core defragmentation functions
> --
>
> Key: IGNITE-13190
> URL: https://issues.apache.org/jira/browse/IGNITE-13190
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Sergey Chugunov
>Assignee: Ivan Bessonov
>Priority: Major
>  Labels: IEP-47
> Fix For: 2.10
>
>  Time Spent: 20h 50m
>  Remaining Estimate: 0h
>
> The following set of functions covering defragmentation happy-case needed:
>  * Initialization of defragmentation manager when node is started in 
> maintenance mode.
>  * Information about partition files is gathered by defrag mgr.
>  * For each partition file corresponding file of defragmented partition is 
> created and initialized.
>  * Keys are transferred from old partitions to new partitions.
>  * Checkpointer is aware of new partition files and flushes defragmented 
> memory to new partition files.
>  
> No fault-tolerance code nor index defragmentation mappings are needed in this 
> task.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13722) Ignite Spring Data: Ignite resources are not closed properly when closing Spring context.

2020-12-14 Thread Mikhail Petrov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249060#comment-17249060
 ] 

Mikhail Petrov commented on IGNITE-13722:
-

[~alex_pl], [~timonin.maksim] Tanks a lot for the review!

> Ignite Spring Data: Ignite resources are not closed properly when closing 
> Spring context.
> -
>
> Key: IGNITE-13722
> URL: https://issues.apache.org/jira/browse/IGNITE-13722
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mikhail Petrov
>Assignee: Mikhail Petrov
>Priority: Major
>
> It's needed to close Ignite thin/thick client that is used by Spring Data for 
> accessing the Ignite cluster when closing Spring context.
> Currently, if Ignite/ClientConfiguration is used to configure Ignite cluster 
> access for a repository (see RepositoryConfig), instances of 
> Ignite/IgniteClient remains unclosed after the close of Spring context.
> To solve this issue it's proposed to move management of Ignite instances 
> needed by Spring Data  for accessing the cluster to separate configurable 
> factory bean. It will be auto-registered by Spring Data and will close all 
> Ignite resources when closing the Spring context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (IGNITE-13792) Reconnecting clients trigger failure handler

2020-12-14 Thread Sergey Uttsel (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Uttsel reassigned IGNITE-13792:
--

Assignee: Sergey Uttsel

> Reconnecting clients trigger failure handler
> 
>
> Key: IGNITE-13792
> URL: https://issues.apache.org/jira/browse/IGNITE-13792
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 2.9
>Reporter: Pavel Vinokurov
>Assignee: Sergey Uttsel
>Priority: Major
> Attachments: UnstableClients.java
>
>
> {code:java}
> Dec 01, 2020 9:38:29 PM java.util.logging.LogManager$RootLogger log
> SEVERE: JVM will be halted immediately due to the failure: 
> [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class 
> o.a.i.IgniteCheckedException: Affinity for topology version is not 
> initialized [locNode=b50635ff-0324-431b-bc34-00a6cd36c9e3, 
> grp=ignite-sys-cache, topVer=AffinityTopologyVersion [topVer=570, 
> minorTopVer=0], head=AffinityTopologyVersion [topVer=569, minorTopVer=0], 
> history=[AffinityTopologyVersion [topVer=551, minorTopVer=0], 
> AffinityTopologyVersion [topVer=552, minorTopVer=0], AffinityTopologyVersion 
> [topVer=553, minorTopVer=0], AffinityTopologyVersion [topVer=554, 
> minorTopVer=0], AffinityTopologyVersion [topVer=555, minorTopVer=0], 
> AffinityTopologyVersion [topVer=556, minorTopVer=0], AffinityTopologyVersion 
> [topVer=557, minorTopVer=0], AffinityTopologyVersion [topVer=558, 
> minorTopVer=0], AffinityTopologyVersion [topVer=559, minorTopVer=0], 
> AffinityTopologyVersion [topVer=560, minorTopVer=0], AffinityTopologyVersion 
> [topVer=561, minorTopVer=0], AffinityTopologyVersion [topVer=562, 
> minorTopVer=0], AffinityTopologyVersion [topVer=563, minorTopVer=0], 
> AffinityTopologyVersion [topVer=564, minorTopVer=0], AffinityTopologyVersion 
> [topVer=565, minorTopVer=0], AffinityTopologyVersion [topVer=566, 
> minorTopVer=0], AffinityTopologyVersion [topVer=567, minorTopVer=0], 
> AffinityTopologyVersion [topVer=568, minorTopVer=0], AffinityTopologyVersion 
> [topVer=569, minorTopVer=0]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13848) Premature update SegmentReservationStorage#minReserveIdx during truncate of segments

2020-12-14 Thread Ignite TC Bot (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249039#comment-17249039
 ] 

Ignite TC Bot commented on IGNITE-13848:


{panel:title=Branch: [pull/8573/head] Base: [master] : No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/8573/head] Base: [master] : New Tests 
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 2{color} [[tests 
2|https://ci.ignite.apache.org/viewLog.html?buildId=5789819]]
* {color:#013220}IgnitePdsTestSuite2: 
IgnitePdsReserveWalSegmentsTest.testMinReserveIdx - PASSED{color}
* {color:#013220}IgnitePdsTestSuite2: 
IgnitePdsReserveWalSegmentsWithCompactionTest.testMinReserveIdx - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5789847buildTypeId=IgniteTests24Java8_RunAll]

> Premature update SegmentReservationStorage#minReserveIdx during truncate of  
> segments
> -
>
> Key: IGNITE-13848
> URL: https://issues.apache.org/jira/browse/IGNITE-13848
> Project: Ignite
>  Issue Type: Bug
>  Components: persistence
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It was found premature *SegmentReservationStorage#minReserveIdx* update in 
> *FileWriteAheadLogManager#truncate*. Which creates the wrong state of the 
> segments in the archive.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13828) Change ducktape dependency to own fork

2020-12-14 Thread Anton Vinogradov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249036#comment-17249036
 ] 

Anton Vinogradov commented on IGNITE-13828:
---

Merged to ignite-ducktape.
Thanks for your contribution.

> Change ducktape dependency to own fork
> --
>
> Key: IGNITE-13828
> URL: https://issues.apache.org/jira/browse/IGNITE-13828
> Project: Ignite
>  Issue Type: Task
>Reporter: Sergei Ryzhov
>Assignee: Sergei Ryzhov
>Priority: Minor
>  Labels: IEP-56, ducktape
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The ducktape repository doesn't contain everything we need, so we need to use 
> our own fork with extensions
> Contains changes:
> Fix cluster property of Test 
> #258 opened on Oct 29 by ivandasch • Review required
> https://github.com/confluentinc/ducktape/pull/258
> Fix BackgroundThreadService constructor. 
> #256 opened on Oct 28 by ivandasch • Review required
> https://github.com/confluentinc/ducktape/pull/256
> Add optional clean parameter to Service.start. 
> #254 opened on Sep 25 by ivandasch • Review required
> https://github.com/confluentinc/ducktape/pull/254
> Fix: set DucktapeJSONEncoder as default encoder in reports. 
> #251 opened on Sep 7 by ivandasch • Review required
> https://github.com/confluentinc/ducktape/pull/251



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13851) Document persistence forward compatibility policy

2020-12-14 Thread Ilya Kasnacheev (Jira)
Ilya Kasnacheev created IGNITE-13851:


 Summary: Document persistence forward compatibility policy
 Key: IGNITE-13851
 URL: https://issues.apache.org/jira/browse/IGNITE-13851
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Ilya Kasnacheev
 Fix For: 2.10


We have a policy that you can upgrade from 2.x to 2.y, y > x, while keeping 
your persistence files.

Let's document that fact.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13847) Make GridEncryptionManager#onWalSegmentRemoved async

2020-12-14 Thread Kirill Tkalenko (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-13847:
-
Description: 
When implementing IGNITE-13831 I was faced with deadlock.

When execute *FileWriteAheadLogManager#rollOver*, begin to clean WAL archive 
since we have reached the *DataStorageConfiguration#maxWalArchiveSize*, after 
deleting a segment, execute the *GridEncryptionManager#onWalSegmentRemoved* 
that wants to write to the metastore, but it will not succeed, since it will 
wait for *FileWriteAheadLogManager#rollOver*.

I suggest making the *GridEncryptionManager#onWalSegmentRemoved* asynchronous 
in a separate pool, for example, as a *CacheGroupPageScanner#singleExecSvc*.

  was:

Make the *GridEncryptionManager#onWalSegmentRemoved* asynchronous to prevent 
deadlocks in the *FileWriteAheadLogManager*.


> Make GridEncryptionManager#onWalSegmentRemoved async
> 
>
> Key: IGNITE-13847
> URL: https://issues.apache.org/jira/browse/IGNITE-13847
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Kirill Tkalenko
>Priority: Major
>  Labels: IEP-18
> Fix For: 2.10
>
>
> When implementing IGNITE-13831 I was faced with deadlock.
> When execute *FileWriteAheadLogManager#rollOver*, begin to clean WAL archive 
> since we have reached the *DataStorageConfiguration#maxWalArchiveSize*, after 
> deleting a segment, execute the *GridEncryptionManager#onWalSegmentRemoved* 
> that wants to write to the metastore, but it will not succeed, since it will 
> wait for *FileWriteAheadLogManager#rollOver*.
> I suggest making the *GridEncryptionManager#onWalSegmentRemoved* asynchronous 
> in a separate pool, for example, as a *CacheGroupPageScanner#singleExecSvc*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13847) Make GridEncryptionManager#onWalSegmentRemoved async

2020-12-14 Thread Kirill Tkalenko (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-13847:
-
Description: 

Make the *GridEncryptionManager#onWalSegmentRemoved* asynchronous to prevent 
deadlocks in the *FileWriteAheadLogManager*.

  was:Make the *GridEncryptionManager#onWalSegmentRemoved* asynchronous to 
prevent deadlocks in the *FileWriteAheadLogManager*.


> Make GridEncryptionManager#onWalSegmentRemoved async
> 
>
> Key: IGNITE-13847
> URL: https://issues.apache.org/jira/browse/IGNITE-13847
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Kirill Tkalenko
>Priority: Major
>  Labels: IEP-18
> Fix For: 2.10
>
>
> Make the *GridEncryptionManager#onWalSegmentRemoved* asynchronous to prevent 
> deadlocks in the *FileWriteAheadLogManager*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13831) Move WAL archive cleanup from checkpoint to rollover

2020-12-14 Thread Kirill Tkalenko (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-13831:
-
Description: 
Users expect *DataStorageConfiguration#maxWalArchiveSize* to mean that WAL 
archive will not exceed this value, but it is not.
It seems that to reduce the chance of getting into a situation when we exceed 
WAL archive, it will be lowed when we clean it when switching to a new segment 
than at the end of the checkpoint. It is proposed to move the archive cleanup 
to *FileWriteAheadLogManager#rollOver* when the 
*DataStorageConfiguration#maxWalArchiveSize* is reached.

  was:
At the moment, WAL archive is cleared at the end of the checkpoint, which does 
not seem correct and needs to be moved to the *FileWriteAheadLogManager*. 

It is suggested to clear WAL archive in *FileWriteAheadLogManager#rollOver* 
when the *DataStorageConfiguration#maxWalArchiveSize* is reached.


> Move WAL archive cleanup from checkpoint to rollover
> 
>
> Key: IGNITE-13831
> URL: https://issues.apache.org/jira/browse/IGNITE-13831
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Users expect *DataStorageConfiguration#maxWalArchiveSize* to mean that WAL 
> archive will not exceed this value, but it is not.
> It seems that to reduce the chance of getting into a situation when we exceed 
> WAL archive, it will be lowed when we clean it when switching to a new 
> segment than at the end of the checkpoint. It is proposed to move the archive 
> cleanup to *FileWriteAheadLogManager#rollOver* when the 
> *DataStorageConfiguration#maxWalArchiveSize* is reached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13850) Improve the TracingSpi interface

2020-12-14 Thread Amelchev Nikita (Jira)
Amelchev Nikita created IGNITE-13850:


 Summary: Improve the TracingSpi interface
 Key: IGNITE-13850
 URL: https://issues.apache.org/jira/browse/IGNITE-13850
 Project: Ignite
  Issue Type: Improvement
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita


Need to investigate and improve the TracingSpi interface:

1. Add support for non-strings tags and log points: primitives, arrays etc
2. Add ability to postpone adding span tags and log points
3. Investigate ability to cache some tags



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (IGNITE-13817) Calcite bug. select count assertionError.

2020-12-14 Thread Konstantin Orlov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov reassigned IGNITE-13817:
-

Assignee: Konstantin Orlov

> Calcite bug. select count assertionError.
> -
>
> Key: IGNITE-13817
> URL: https://issues.apache.org/jira/browse/IGNITE-13817
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Affects Versions: 2.9
>Reporter: Stanilovsky Evgeny
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: calcite
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> request:
> {noformat}
> SELECT count(*)
> FROM 
>  R,
>  T,
>  B
> WHERE R.BKEY = B.KEY
> AND R.ID = T.ID
> AND R.VERSION = T.VERSION
> AND T.BOOK = ''
> AND B.ISL = TRUE
> {noformat}
> breaks down calcite:
> {noformat}
> java.lang.AssertionError
>   at org.apache.calcite.plan.volcano.RelSet.mergeWith(RelSet.java:457)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.merge(VolcanoPlanner.java:1046)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.rename(VolcanoPlanner.java:903)
>   at org.apache.calcite.plan.volcano.RelSet.mergeWith(RelSet.java:432)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.merge(VolcanoPlanner.java:1046)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerSubset(VolcanoPlanner.java:1287)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1166)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:589)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:604)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:148)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:268)
>   at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:283)
>   at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:169)
>   at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:229)
>   at 
> org.apache.calcite.plan.volcano.TopDownRuleDriver.applyGenerator(TopDownRuleDriver.java:142)
>   at 
> org.apache.calcite.plan.volcano.TopDownRuleDriver.access$600(TopDownRuleDriver.java:47)
>   at 
> org.apache.calcite.plan.volcano.TopDownRuleDriver$ApplyRule.perform(TopDownRuleDriver.java:519)
>   at 
> org.apache.calcite.plan.volcano.TopDownRuleDriver.drive(TopDownRuleDriver.java:101)
>   at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
>   at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:312)
>   at 
> org.apache.ignite.internal.processors.query.calcite.prepare.IgnitePlanner.transform(IgnitePlanner.java:258)
>   at 
> org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.optimize(ExecutionServiceImpl.java:616)
>   at 
> org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepareQuery(ExecutionServiceImpl.java:568)
>   at 
> org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepareSingle(ExecutionServiceImpl.java:542)
>   at 
> org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.prepareQuery(ExecutionServiceImpl.java:501)
>   at 
> org.apache.ignite.internal.processors.query.calcite.prepare.QueryPlanCacheImpl.queryPlan(QueryPlanCacheImpl.java:84)
>   at 
> org.apache.ignite.internal.processors.query.calcite.exec.ExecutionServiceImpl.executeQuery(ExecutionServiceImpl.java:378)
>   at 
> org.apache.ignite.internal.processors.query.calcite.CalciteQueryProcessor.query(CalciteQueryProcessor.java:240)
>   at 
> org.apache.ignite.internal.processors.query.calcite.CalciteQueryProcessorTest.test0(CalciteQueryProcessorTest.java:117)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2373)
>   at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This 

[jira] [Commented] (IGNITE-13734) .NET Service loses returned array type information

2020-12-14 Thread Yaroslav Molochkov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248895#comment-17248895
 ] 

Yaroslav Molochkov commented on IGNITE-13734:
-

[~kukushal], I think we need to investigate .net ability to cast results to the 
needed type, so it might turn out that this ticket is not needed

> .NET Service loses returned array type information
> --
>
> Key: IGNITE-13734
> URL: https://issues.apache.org/jira/browse/IGNITE-13734
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Affects Versions: 2.9
>Reporter: Alexey Kukushkin
>Assignee: Nikolai Kulagin
>Priority: Major
>
> .NET service client receives Object[] instead of strongly typed array from a 
> .NET service. 
> There was another already resolved similar issue IGNITE-12823 that addressed 
> the problem of using arrays as parameters. The problem of using arrays as 
> results still exists.
> h3. Reproducer
> A .NET service returning an array of user-defined types is deployed:
> {code:c#}
> public interface ITestService
> {
> Parameter[] TestReturnParametersArray();
> }
> public sealed class Parameter
> {
> public int Id { get; set; }
> public int[] Values { get; set; }
> }
> {code}
> A .NET client calls the service:
> {code:c#}
> Parameter[] res = svcProxy.TestReturnParametersArray()
> {code}
> The service call fails with exception:
> {code}
> System.InvalidCastException : Unable to cast object of type 'System.Object[]' 
> to type 'Parameter[]'.
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13748) Schema configuration public API

2020-12-14 Thread Yury Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yury Gerzhedovich updated IGNITE-13748:
---
Component/s: sql

> Schema configuration public API
> ---
>
> Key: IGNITE-13748
> URL: https://issues.apache.org/jira/browse/IGNITE-13748
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Andrey Mashenkov
>Assignee: Andrey Mashenkov
>Priority: Major
>  Labels: iep-54, ignite-3
>
> Let's implement a public API classes for initial schema declaration. 
> Considering next:
>  * It should be possible to create schema without having any Class instance, 
> like we do now with QueryEntity. 
>  * Automatic schema creation based on annotated classes, like we have in 
> ccfg.setIndexedTypes()
>  * Transient fields semantic.
>  * Using builder pattern for schema configuration.
>  * Split type-system into internal native-types (introduced GG-13617) and 
> public API types. The latest ones should be portable (e.g. could be used in 
> .Net)
>  * API can be extracted to separate module that will not have any dependency 
> on other Ignite modules.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-13849) Calcite Integration. Add unit tests for all IgniteRel nodes serialization / deserialization

2020-12-14 Thread Konstantin Orlov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248885#comment-17248885
 ] 

Konstantin Orlov commented on IGNITE-13849:
---

One of the possible issue here - missed collation of IgniteExchange

> Calcite Integration. Add unit tests for all IgniteRel nodes serialization / 
> deserialization
> ---
>
> Key: IGNITE-13849
> URL: https://issues.apache.org/jira/browse/IGNITE-13849
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Reporter: Taras Ledkov
>Assignee: Taras Ledkov
>Priority: Major
>
> We have to check serialization for each node (child classes of the IgniteRel).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-12892) Clarify WAL archive size configuration

2020-12-14 Thread Kirill Tkalenko (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248872#comment-17248872
 ] 

Kirill Tkalenko commented on IGNITE-12892:
--

Hi, [~shm]! 
I think you can ask the release engineer to add this ticket to the 
[release|https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9.1#ApacheIgnite2.9.1-2.9/2.9.1PerformanceComparison].

It seems to me that most likely the IGNITE-13373 fixed the problem you 
indicated, I already made a description in the [PR's comment 
|https://github.com/apache/ignite/pull/8550#issuecomment-744032357].





> Clarify WAL archive size configuration
> --
>
> Key: IGNITE-12892
> URL: https://issues.apache.org/jira/browse/IGNITE-12892
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Actual maximum size of WAL archive that can be reserved for historical 
> rebalance is calculated as minimum of three properties:
>  # DataStorageConfiguration#walHistSize (units: number of checkpoints)
>  # DataStorageConfiguration#maxWalArchiveSize (units: bytes)
>  # IgniteSystemProperties#IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE 
> (units: number of checkpoints)
> The logic is a little unclear, so I propose following changes:
>  # Stop using walHistSize at all (it's already deprecated) for WAL truncation
>  # Use IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE only in case WAL archive 
> is managed externally (so we limit the quantity of checkpoints stored in 
> memory, but don't remove WAL files)
>  # Use -1 for maxWalArchiveSize (instead of Long.MAX_VALUE) to disable WAL 
> truncation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13849) Calcite Integration. Add unit tests for all IgniteRel nodes serialization / deserialization

2020-12-14 Thread Taras Ledkov (Jira)
Taras Ledkov created IGNITE-13849:
-

 Summary: Calcite Integration. Add unit tests for all IgniteRel 
nodes serialization / deserialization
 Key: IGNITE-13849
 URL: https://issues.apache.org/jira/browse/IGNITE-13849
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


We have to check serialization for each node (child classes of the IgniteRel).




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (IGNITE-12892) Clarify WAL archive size configuration

2020-12-14 Thread shivakumar (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248845#comment-17248845
 ] 

shivakumar edited comment on IGNITE-12892 at 12/14/20, 9:46 AM:


Is this code change part of 2.9.1 release ? 

As per my testing looks like this code change solves the blocking issue 
reported here

[http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html]

[http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html]

 

If it is not part of 2.9.1 then is it possible to backport to 2.9.0 easily ? Or 
it has dependency on other code changes?

 


was (Author: shm):
Is this code change part of 2.9.1 release ? 

As pery testing looks like this code change solves the blocking issue reported 
here

[http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html]

[http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html]

 

> Clarify WAL archive size configuration
> --
>
> Key: IGNITE-12892
> URL: https://issues.apache.org/jira/browse/IGNITE-12892
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Actual maximum size of WAL archive that can be reserved for historical 
> rebalance is calculated as minimum of three properties:
>  # DataStorageConfiguration#walHistSize (units: number of checkpoints)
>  # DataStorageConfiguration#maxWalArchiveSize (units: bytes)
>  # IgniteSystemProperties#IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE 
> (units: number of checkpoints)
> The logic is a little unclear, so I propose following changes:
>  # Stop using walHistSize at all (it's already deprecated) for WAL truncation
>  # Use IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE only in case WAL archive 
> is managed externally (so we limit the quantity of checkpoints stored in 
> memory, but don't remove WAL files)
>  # Use -1 for maxWalArchiveSize (instead of Long.MAX_VALUE) to disable WAL 
> truncation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (IGNITE-12892) Clarify WAL archive size configuration

2020-12-14 Thread shivakumar (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248845#comment-17248845
 ] 

shivakumar commented on IGNITE-12892:
-

Is this code change part of 2.9.1 release ? 

As pery testing looks like this code change solves the blocking issue reported 
here

[http://apache-ignite-users.70518.x6.nabble.com/Could-not-clear-historyMap-due-to-WAL-reservation-on-cp-td34779.html]

[http://apache-ignite-users.70518.x6.nabble.com/connecting-to-visor-shell-permanently-stops-unwanted-WAL-clean-up-td34737.html]

 

> Clarify WAL archive size configuration
> --
>
> Key: IGNITE-12892
> URL: https://issues.apache.org/jira/browse/IGNITE-12892
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Actual maximum size of WAL archive that can be reserved for historical 
> rebalance is calculated as minimum of three properties:
>  # DataStorageConfiguration#walHistSize (units: number of checkpoints)
>  # DataStorageConfiguration#maxWalArchiveSize (units: bytes)
>  # IgniteSystemProperties#IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE 
> (units: number of checkpoints)
> The logic is a little unclear, so I propose following changes:
>  # Stop using walHistSize at all (it's already deprecated) for WAL truncation
>  # Use IGNITE_PDS_MAX_CHECKPOINT_MEMORY_HISTORY_SIZE only in case WAL archive 
> is managed externally (so we limit the quantity of checkpoints stored in 
> memory, but don't remove WAL files)
>  # Use -1 for maxWalArchiveSize (instead of Long.MAX_VALUE) to disable WAL 
> truncation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (IGNITE-13848) Premature update SegmentReservationStorage#minReserveIdx during truncate of segments

2020-12-14 Thread Kirill Tkalenko (Jira)
Kirill Tkalenko created IGNITE-13848:


 Summary: Premature update SegmentReservationStorage#minReserveIdx 
during truncate of  segments
 Key: IGNITE-13848
 URL: https://issues.apache.org/jira/browse/IGNITE-13848
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.10


It was found premature *SegmentReservationStorage#minReserveIdx* update in 
*FileWriteAheadLogManager#truncate*. Which creates the wrong state of the 
segments in the archive.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Comment Edited] (IGNITE-13618) Provide generated and reflection-based class (de)serializers

2020-12-14 Thread Andrey Mashenkov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248442#comment-17248442
 ] 

Andrey Mashenkov edited comment on IGNITE-13618 at 12/14/20, 9:07 AM:
--

1. Unfortunatelly, VarHandles/MethodHandle support is limited in Janino.
Seems, it can't correctly compile code and link MethodHandler invoke methods 
with polymorphic signatures. 

As we can't write plain code and access fields/call method directly therefore 
generated code will be very similar to the reflection-based code and 
hardly possible it could be more efficient.

2. Janino expects class body code only, but JavaPoet can generate only full 
class code with class signature and package declaration.


was (Author: amashenkov):
1. Unfortunatelly, VarHandles/MethodHandle support is limited in Janino.
Seems, it can't correctly compile code and link MethodHandler invoke methods 
with polymorphic signatures. 

As we can't write plain code and access fields/call method directly therefore 
generated code will be very similar to the reflection-based code and 
hardly possible it could be more efficient.

2. Janino expects class body code only, but however JavaPoet can generate only 
full class code with class signature and package declaration.

> Provide generated and reflection-based class (de)serializers
> 
>
> Key: IGNITE-13618
> URL: https://issues.apache.org/jira/browse/IGNITE-13618
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Alexey Goncharuk
>Assignee: Andrey Mashenkov
>Priority: Major
>  Labels: iep-54, ignite-3
> Attachments: benchmark-results.txt
>
>
> Let's prototype object serializer for type-system described in IEP-54, and 
> benchmark them to check if generated code approach is a better one.
>  * For reflection-based serializer we can use Unsafe (see GridUnsafe class) 
> which works fine for previour versions of Ignite.
>  * To keep focus on the ticket needs mainly, we can use Janino project for 
> code generator.
>  * In addition, reflection-based serialized could have a version using Java9+ 
> VanHandles and it's could be hidden under 'java-9+' maven-profile.
> Hints:
> Javapoet can be used as codegenerator, Janino as a compiler, class loader. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13659) TDE - Phase 3. Documentation

2020-12-14 Thread Pavel Pereslegin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Pereslegin updated IGNITE-13659:
--
Attachment: (was: html-ver.zip)

> TDE - Phase 3. Documentation
> 
>
> Key: IGNITE-13659
> URL: https://issues.apache.org/jira/browse/IGNITE-13659
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Pavel Pereslegin
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: IEP-18, tde-phase-3
> Fix For: 2.10
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Write documentation on the cache encryption key rotation function.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (IGNITE-13845) Add checkpoint lock hold metrics

2020-12-14 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-13845:
-
Fix Version/s: 2.10

> Add checkpoint lock hold metrics
> 
>
> Key: IGNITE-13845
> URL: https://issues.apache.org/jira/browse/IGNITE-13845
> Project: Ignite
>  Issue Type: New Feature
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
> Fix For: 2.10
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add last checkpoint lock hold duration and durations histogram metrics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)