[
https://issues.apache.org/jira/browse/IGNITE-21356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-21356:
-------------------------------------
Description:
{code:java}
[2024-01-25T10:28:50,418][INFO ][%sqllogic1%checkpoint-thread-4][Checkpointer]
Skipping checkpoint (no pages were modified)
[checkpointBeforeWriteLockTime=0ms, checkpointWriteLockWait=0ms,
checkpointListenersExecuteTime=0ms, checkpointWriteLockHoldTime=0ms,
reason='timeout']------- Stderr: -------[2024-01-25T10:20:17,371][WARN
][%sqllogic0%metastorage-watch-executor-0][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1342,
catalog.version]][2024-01-25T10:20:17,374][WARN
][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1342,
catalog.version]][2024-01-25T10:20:18,268][WARN
][%sqllogic0%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=238, keys=[catalog.update.1343,
catalog.version]][2024-01-25T10:20:18,269][WARN
][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1343,
catalog.version]][2024-01-25T10:20:19,117][WARN
][%sqllogic0%tableManager-io-7][CatalogManagerImpl] Failed to apply catalog
update.java.util.concurrent.CompletionException:
org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException:
IGN-CMN-65535 TraceId:061dbe67-7b05-4773-957a-d568db1c5897 Out of memory in
data region [name=default, size=256.0 MiB, persistence=true] Try the following:
^-- Increase maximum off-heap memory size
(PersistentPageMemoryDataRegionConfigurationSchema.size) ^-- Enable eviction
or expiration policies at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
~[?:?] at
java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683)
~[?:?] at
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
~[?:?] at
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
~[?:?] at
org.apache.ignite.internal.storage.util.MvPartitionStorages.create(MvPartitionStorages.java:120)
~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?] at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.lambda$createMvPartition$5(AbstractPageMemoryTableStorage.java:163)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
...
Caused by:
org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException: Failed to
find a page for eviction (no pages to replace) [segmentCapacity=6805,
loaded=2709, dirtyPagesSoftThreshold=2031, dirtyPagesHardThreshold=2438,
dirtyPages=2709, pinned=0]{code}
[https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7799962?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true&expandBuildChangesSection=true]
There's also suspicious
{code:java}
Caused by: java.lang.AssertionError: FullPageId [pageId=000100000000000c,
effectivePageId=000000000000000c, groupId=770]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:758)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:641)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:613)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:280)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:296)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.freelist.PagesList.flushBucketsCache(PagesList.java:387)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.freelist.PagesList.saveMetadata(PagesList.java:332)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.mv.RowVersionFreeList.saveMetadata(RowVersionFreeList.java:185)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$syncMetadataOnCheckpoint$13(PersistentPageMemoryMvPartitionStorage.java:345)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.checkpoint.AwaitTasksCompletionExecutor.lambda$execute$1(AwaitTasksCompletionExecutor.java:59)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
... 1 more {code}
was:
{code:java}
[2024-01-25T10:28:50,418][INFO ][%sqllogic1%checkpoint-thread-4][Checkpointer]
Skipping checkpoint (no pages were modified)
[checkpointBeforeWriteLockTime=0ms, checkpointWriteLockWait=0ms,
checkpointListenersExecuteTime=0ms, checkpointWriteLockHoldTime=0ms,
reason='timeout']------- Stderr: -------[2024-01-25T10:20:17,371][WARN
][%sqllogic0%metastorage-watch-executor-0][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1342,
catalog.version]][2024-01-25T10:20:17,374][WARN
][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1342,
catalog.version]][2024-01-25T10:20:18,268][WARN
][%sqllogic0%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=238, keys=[catalog.update.1343,
catalog.version]][2024-01-25T10:20:18,269][WARN
][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
processing has been too long [duration=236, keys=[catalog.update.1343,
catalog.version]][2024-01-25T10:20:19,117][WARN
][%sqllogic0%tableManager-io-7][CatalogManagerImpl] Failed to apply catalog
update.java.util.concurrent.CompletionException:
org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException:
IGN-CMN-65535 TraceId:061dbe67-7b05-4773-957a-d568db1c5897 Out of memory in
data region [name=default, size=256.0 MiB, persistence=true] Try the following:
^-- Increase maximum off-heap memory size
(PersistentPageMemoryDataRegionConfigurationSchema.size) ^-- Enable eviction
or expiration policies at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
~[?:?] at
java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683)
~[?:?] at
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
~[?:?] at
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
~[?:?] at
org.apache.ignite.internal.storage.util.MvPartitionStorages.create(MvPartitionStorages.java:120)
~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?] at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.lambda$createMvPartition$5(AbstractPageMemoryTableStorage.java:163)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
...
Caused by:
org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException: Failed to
find a page for eviction (no pages to replace) [segmentCapacity=6805,
loaded=2709, dirtyPagesSoftThreshold=2031, dirtyPagesHardThreshold=2438,
dirtyPages=2709, pinned=0]{code}
https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7799962?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true&expandBuildChangesSection=true
> ItSqlLogicTest sql(types/decimal/test_decimal_ops.test) failed with
> IgniteOutOfMemoryException
> ----------------------------------------------------------------------------------------------
>
> Key: IGNITE-21356
> URL: https://issues.apache.org/jira/browse/IGNITE-21356
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexander Lapin
> Priority: Major
> Labels: MakeTeamcityGreenAgain, ignite-3
>
> {code:java}
> [2024-01-25T10:28:50,418][INFO
> ][%sqllogic1%checkpoint-thread-4][Checkpointer] Skipping checkpoint (no pages
> were modified) [checkpointBeforeWriteLockTime=0ms,
> checkpointWriteLockWait=0ms, checkpointListenersExecuteTime=0ms,
> checkpointWriteLockHoldTime=0ms, reason='timeout']------- Stderr:
> -------[2024-01-25T10:20:17,371][WARN
> ][%sqllogic0%metastorage-watch-executor-0][WatchProcessor] Watch event
> processing has been too long [duration=236, keys=[catalog.update.1342,
> catalog.version]][2024-01-25T10:20:17,374][WARN
> ][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
> processing has been too long [duration=236, keys=[catalog.update.1342,
> catalog.version]][2024-01-25T10:20:18,268][WARN
> ][%sqllogic0%metastorage-watch-executor-1][WatchProcessor] Watch event
> processing has been too long [duration=238, keys=[catalog.update.1343,
> catalog.version]][2024-01-25T10:20:18,269][WARN
> ][%sqllogic1%metastorage-watch-executor-1][WatchProcessor] Watch event
> processing has been too long [duration=236, keys=[catalog.update.1343,
> catalog.version]][2024-01-25T10:20:19,117][WARN
> ][%sqllogic0%tableManager-io-7][CatalogManagerImpl] Failed to apply catalog
> update.java.util.concurrent.CompletionException:
> org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException:
> IGN-CMN-65535 TraceId:061dbe67-7b05-4773-957a-d568db1c5897 Out of memory in
> data region [name=default, size=256.0 MiB, persistence=true] Try the
> following: ^-- Increase maximum off-heap memory size
> (PersistentPageMemoryDataRegionConfigurationSchema.size) ^-- Enable eviction
> or expiration policies at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> ~[?:?] at
> java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:683)
> ~[?:?] at
> java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
> ~[?:?] at
> java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
> ~[?:?] at
> org.apache.ignite.internal.storage.util.MvPartitionStorages.create(MvPartitionStorages.java:120)
> ~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?] at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.lambda$createMvPartition$5(AbstractPageMemoryTableStorage.java:163)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> ...
> Caused by:
> org.apache.ignite.internal.pagememory.mem.IgniteOutOfMemoryException: Failed
> to find a page for eviction (no pages to replace) [segmentCapacity=6805,
> loaded=2709, dirtyPagesSoftThreshold=2031, dirtyPagesHardThreshold=2438,
> dirtyPages=2709, pinned=0]{code}
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7799962?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true&expandBuildChangesSection=true]
>
> There's also suspicious
> {code:java}
> Caused by: java.lang.AssertionError: FullPageId [pageId=000100000000000c,
> effectivePageId=000000000000000c, groupId=770]
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:758)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:641)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.acquirePage(PersistentPageMemory.java:613)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:280)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:296)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.freelist.PagesList.flushBucketsCache(PagesList.java:387)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.freelist.PagesList.saveMetadata(PagesList.java:332)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.mv.RowVersionFreeList.saveMetadata(RowVersionFreeList.java:185)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$syncMetadataOnCheckpoint$13(PersistentPageMemoryMvPartitionStorage.java:345)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.pagememory.persistence.checkpoint.AwaitTasksCompletionExecutor.lambda$execute$1(AwaitTasksCompletionExecutor.java:59)
> ~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> ... 1 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
