[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901302#comment-16901302
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/8/19 10:34 AM:
---
Thank you, Pavel, for the review!
So, I would propose to change the formula as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.pageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.pageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
was (Author: slava.koptilin):
Thank you, Pavel, for the review!
So, I would propose to change the formula as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901302#comment-16901302
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 5:23 PM:
--
Thank you, Pavel, for the review!
So, I propose to change the formula as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
was (Author: slava.koptilin):
Thank you, Pavel, for the review!
So, I propose to change the equation as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901302#comment-16901302
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 5:23 PM:
--
Thank you, Pavel, for the review!
So, I would propose to change the formula as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
was (Author: slava.koptilin):
Thank you, Pavel, for the review!
So, I propose to change the formula as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901302#comment-16901302
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 5:22 PM:
--
Thank you, Pavel, for the review!
So, I propose to change the equation as follows:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /one page per bucket/);{code}
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
was (Author: slava.koptilin):
Thank you, Pavel, for the review!
So, I propose to change the equation as follows:
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 * 1.5 /
pageMem.pageSize() + 1) + 256 /*one page per bucket*/);
where:
{{memorySize / pageMem.systemPageSize()) - total pages}}
{{datarow.size() / pageMem.systemPageSize() - data row pages}}
{{nonEmptyPages * (8.0 * 1.5 / pageMem.pageSize() + 1) - nonEmptyPages +
linkPages (multiplied by 1.5)}}
{{256 - additional pages reserved for buckets of FreeList}}
{{What do you think, [~xtern] [~ivan.glukos]? I am ready to update the PR and
re-run TC. :)}}
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901250#comment-16901250 ] Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 4:39 PM: --- [~slava.koptilin], > _Hmm... nonEmptyPages * (coeff + 1) = nonEmptyPages * coef + nonEmptyPages = > linkPages + nonEmptyPages_ Sorry, my bad, I missed +1. Thanks for explanation. was (Author: xtern): [~slava.koptilin], _Hmm... nonEmptyPages * (coeff + 1) = nonEmptyPages * coef + nonEmptyPages = linkPages + nonEmptyPages_ Sorry, my bad, I missed +1. Thanks for explanation. > Atempt to remove entries from fully populated data region may result in > IgineOutOfMemoryException > - > > Key: IGNITE-12042 > URL: https://issues.apache.org/jira/browse/IGNITE-12042 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Vyacheslav Koptilin >Assignee: Vyacheslav Koptilin >Priority: Major > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Removing entries from non-persistent data region may require allocating a new > data page in order to move a tracked page from one bucket of the free-list to > another one. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901152#comment-16901152
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 3:11 PM:
--
[~xtern]
> Or am I missing something?
Yep, we are calculating total pages. To be more precise:
{{datarowPages}} = {{datarow.size() / pageMem.systemPageSize()}}
{{linkPages + nonEmptyPages }}= {{nonEmptyPages * (8.0 / pageMem.pageSize() +
*1*)}}
{{additionalPages}} = {{256 // one page per bucket}}
So,
*totalPages* < {{datarowPages + linkPages + nonEmptyPages + additionalPages}}
> AFAIK each data page has overhead
{{> (see AbstractDataPageIO.MIN_DATA_PAGE_OVERHEAD, fragmented data row
required 8 bytes more on each page)}}
{{> therefore, these calculations for data row are very approximate.}}
That is the reason why we have the multiplier - 1.5
was (Author: slava.koptilin):
[~xtern]
> Or am I missing something?
Yep, we are calculating total pages. To be more precise:
{{datarowPages}} = {{datarow.size() / pageMem.systemPageSize()}}
{{linkPages + nonEmptyPages }}= {{nonEmptyPages * (8.0 / pageMem.pageSize() +
*1*)}}
{{additionalPages}} = {{256 // one page per bucket}}
So,
*totalPages* < {{datarowPages + linkPages + nonEmptyPages + additionalPages}}
{\{> AFAIK each data page has overhead }}
{{> (see AbstractDataPageIO.MIN_DATA_PAGE_OVERHEAD, fragmented data row
required 8 bytes more on each page)}}
{{> therefore, these calculations for data row are very approximate.}}
That is the reason why we have the multiplier - 1.5
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901152#comment-16901152
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 3:10 PM:
--
[~xtern]
> Or am I missing something?
Yep, we are calculating total pages. To be more precise:
{{datarowPages}} = {{datarow.size() / pageMem.systemPageSize()}}
{{linkPages + nonEmptyPages }}= {{nonEmptyPages * (8.0 / pageMem.pageSize() +
*1*)}}
{{additionalPages}} = {{256 // one page per bucket}}
So,
*totalPages* < {{datarowPages + linkPages + nonEmptyPages + additionalPages}}
{\{> AFAIK each data page has overhead }}
{{> (see AbstractDataPageIO.MIN_DATA_PAGE_OVERHEAD, fragmented data row
required 8 bytes more on each page)}}
{{> therefore, these calculations for data row are very approximate.}}
That is the reason why we have the multiplier - 1.5
was (Author: slava.koptilin):
[~xtern]
> Or am I missing something?
Yep, we are calculating total pages. To be more precise:
{{datarowPages}} = {{datarow.size() / pageMem.systemPageSize()}}
{{linkPages + nonEmptyPages }}= {{nonEmptyPages * (8.0 / pageMem.pageSize() +
*1*)}}
{{additionalPages}} = 256 // one page per bucket
So,
*totalPages* < {{datarowPages + linkPages + nonEmptyPages + additionalPages}}
{{> AFAIK each data page has overhead }}
{{> (see AbstractDataPageIO.MIN_DATA_PAGE_OVERHEAD, fragmented data row
required 8 bytes more on each page)}}
{{> therefore, these calculations for data row are very approximate.}}
That is the reason why we have the multiplier - 1.5
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900999#comment-16900999
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/6/19 1:05 PM:
--
> If the user uses default failure handler all these cases are unrecoverable.
Yep, we are talking about the case when a custom failure handler is used.
> Check will not catch a problem (if datarow is big enough) - throwing OOME
>from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
In my opinion, this should be avoided. We need to take into account the size
of the data row:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 /
pageMem.pageSize() + 1) * 1.5 + 256 /*one page per bucket*/);
{code}
WDYT?
(By the way, the initial goal of this change is reserving pages for internal
data structures: {{PagesListNodeIO}}, {{PagesListMetaIO}})
was (Author: slava.koptilin):
> If the user uses default failure handler all these cases are unrecoverable.
Yep, we are talking about the case when a custom failure handler is used.
> Check will not catch a problem (if datarow is big enough) - throwing OOME
>from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
In my opinion, this should be avoided. We need to take into account the size of
the data row:
{code:java}
boolean oomThreshold = (memorySize / pageMem.systemPageSize()) <
(datarow.size() / pageMem.systemPageSize() + nonEmptyPages * (8.0 /
pageMem.pageSize() + 1) * 1.5 + 256 /*one page per bucket*/);
{code}
WDYT?
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900995#comment-16900995 ] Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 12:46 PM: [~slava.koptilin], I think it would be even better to perform this check from PageMemoryNoStoreImpl#allocatePage for flags = FLAG_DATA, if possible. WDYT? was (Author: xtern): [~slava.koptilin], I think it would be even better to perform this check from PageMemoryImpl#allocatePage for flags = FLAG_DATA, if possible. WDYT? > Atempt to remove entries from fully populated data region may result in > IgineOutOfMemoryException > - > > Key: IGNITE-12042 > URL: https://issues.apache.org/jira/browse/IGNITE-12042 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Vyacheslav Koptilin >Assignee: Vyacheslav Koptilin >Priority: Major > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Removing entries from non-persistent data region may require allocating a new > data page in order to move a tracked page from one bucket of the free-list to > another one. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 12:08 PM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see significant difference here, even with the proposed patch we can
prepare really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If the user configures custom failure handler case 1 and 2 are recoverable.
With 2 a memory leak is possible, but how can a user understand what was the
case?
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see significant difference here, even with the proposed patch we can
prepare really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If the user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 12:02 PM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see significant difference here, even with the proposed patch we can
prepare really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If the user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only
> one page was written, for example, and after that OutOfMemoryException was
> thrown by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME
from DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If the user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 11:54 AM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only
> one page was written, for example, and after that OutOfMemoryException was
> thrown by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME
from DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If the user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 11:53 AM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {{ensureFreeSpaceForInsert}} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {{PageMemoryNoStoreImpl#allocatePage}} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{{PageMemoryNoStoreImpl#allocatePage}}, but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {ensureFreeSpaceForInsert} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {PageMemoryNoStoreImpl#allocatePage} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{PageMemoryNoStoreImpl#allocatePage}, but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 11:51 AM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see the difference here, even with the proposed patch we can prepare
really large data rows and will go in one of the following situations:
1. Check {ensureFreeSpaceForInsert} will catch a problem - throwing OOME from
DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if data row is big enough) - throwing OOME
from {PageMemoryNoStoreImpl#allocatePage} (entry is partially written).
3. Check will not catch a problem (if data row is big enough) - no OOME in
{PageMemoryNoStoreImpl#allocatePage}, but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but
with 2 a memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only
> one page was written, for example, and after that OutOfMemoryException was
> thrown by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see difference here, even with the proposed patch we can prepare really
large data rows and will go in one of the following situations:
1. Check (\{ensureFreeSpaceForInsert}) will catch a problem - throwing OOME
from DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if datarow is big enough) - throwing OOME
from
{PageMemoryNoStoreImpl#allocatePage} (entry is partially written).
3. Check will not catch a problem (if datarow is big enough) - no OOME in
\{PageMemoryNoStoreImpl#allocatePage}
but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but a
memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900927#comment-16900927
]
Pavel Pereslegin edited comment on IGNITE-12042 at 8/6/19 11:48 AM:
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only
> one page was written, for example, and after that OutOfMemoryException was
> thrown by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see difference here, even with the proposed patch we can prepare really
large data rows and will go in one of the following situations:
1. Check (\{ensureFreeSpaceForInsert}) will catch a problem - throwing OOME
from DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if datarow is big enough) - throwing OOME
from
{PageMemoryNoStoreImpl#allocatePage} (entry is partially written).
3. Check will not catch a problem (if datarow is big enough) - no OOME in
\{PageMemoryNoStoreImpl#allocatePage}
but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but a
memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
was (Author: xtern):
[~slava.koptilin],
> _Let's assume the size of an entry is greater than the page size and only one
> page was written, for example, and after that OutOfMemoryException was thrown
> by allocateDataPage()._
> _So, we have partially-written entry, and I am not sure it is a recoverable
> case._
I don't see difference here, even with the proposed patch we can prepare really
large data rows and will go in one of the following situations:
1. Check ({ensureFreeSpaceForInsert}}) will catch a problem - throwing OOME
from DatabaseSharedManager#ensureFreeSpaceForInsert.
2. Check will not catch a problem (if datarow is big enough) - throwing OOME
from {PageMemoryNoStoreImpl#allocatePage} (entry is partially written).
3. Check will not catch a problem (if datarow is big enough) - no OOME in
{PageMemoryNoStoreImpl#allocatePage} but we got OOME when clear() is called.
If a user configures custom failure handler case 1 and 2 are recoverable, but a
memory leak is possible.
If the user uses default failure handler all these cases are unrecoverable.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[
https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900475#comment-16900475
]
Vyacheslav Koptilin edited comment on IGNITE-12042 at 8/5/19 11:34 PM:
---
Hello [~xtern],
First of all, thanks for your interest in this issue! :)
The test should catch the case when removing an entry(persistence and page
eviction should be disabled as well) requires allocating a new data page
(\{{PagesListNodeIO}}) in order to move the tracked page from one bucket of
free-list to another, and there is not enough space for that (fully exhausted
data region).
So, you need to play with the maximum size of the data region, the number of
entries and the size of the entry.
For instance, {{byte[4000]}} works to me. I have updated the PR accordingly.
was (Author: slava.koptilin):
Hello [~xtern],
First of all, thanks for your interest in this issue! :)
The test should catch the case when removing an entry(persistence and page
eviction should be disabled as well) requires allocating a new dat page
(PagesListNodeIO) in order to move the tracked page from one bucket of
free-list to another, and there is not enough space for that (fully exhausted
data region).
So, you need to play with the maximum size of the data region, the number of
entries and the size of the entry.
For instance, byte[4000] works to me. I have updated the PR accordingly.
> Atempt to remove entries from fully populated data region may result in
> IgineOutOfMemoryException
> -
>
> Key: IGNITE-12042
> URL: https://issues.apache.org/jira/browse/IGNITE-12042
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Vyacheslav Koptilin
>Assignee: Vyacheslav Koptilin
>Priority: Major
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Removing entries from non-persistent data region may require allocating a new
> data page in order to move a tracked page from one bucket of the free-list to
> another one.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (IGNITE-12042) Atempt to remove entries from fully populated data region may result in IgineOutOfMemoryException
[ https://issues.apache.org/jira/browse/IGNITE-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900230#comment-16900230 ] Pavel Pereslegin edited comment on IGNITE-12042 at 8/5/19 6:44 PM: --- [~slava.koptilin], I'm trying to understand the problem that this patch solves. The description says that we can get OOME when clear() is called, but the included test (CacheIgniteOutOfMemoryExceptionTest) doesn't fails on current master-branch - is it ok? was (Author: xtern): [~slava.koptilin], I’m trying to understand these changes and started by checking the included test (CacheIgniteOutOfMemoryExceptionTest). This test passes on current master-branch - is it ok? > Atempt to remove entries from fully populated data region may result in > IgineOutOfMemoryException > - > > Key: IGNITE-12042 > URL: https://issues.apache.org/jira/browse/IGNITE-12042 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Vyacheslav Koptilin >Assignee: Vyacheslav Koptilin >Priority: Major > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Removing entries from non-persistent data region may require allocating a new > data page in order to move a tracked page from one bucket of the free-list to > another one. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
