[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2020-12-29 Thread Maxim Muzafarov (Jira) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maxim Muzafarov updated IGNITE-10015:
-
Fix Version/s: (was: 2.10)
   (was: 3.0)

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
>Priority: Critical
> Attachments: hs_err_pid9126.log
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2020-06-29 Thread Aleksey Plekhanov (Jira) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Plekhanov updated IGNITE-10015:
---
Fix Version/s: (was: 2.9)
   2.10

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
>Priority: Critical
> Fix For: 3.0, 2.10
>
> Attachments: hs_err_pid9126.log
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2019-10-02 Thread Maxim Muzafarov (Jira) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maxim Muzafarov updated IGNITE-10015:
-
Fix Version/s: (was: 2.8)
   2.9
   3.0

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
>Priority: Critical
> Fix For: 3.0, 2.9
>
> Attachments: hs_err_pid9126.log
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bu

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2018-11-02 Thread Vladimir Ozerov (JIRA) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-10015:
-
Fix Version/s: (was: 2.7)
   2.8

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Assignee: Sergey Kozlov
>Priority: Critical
> Fix For: 2.8
>
> Attachments: hs_err_pid9126.log
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/c

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2018-10-26 Thread Sergey Kozlov (JIRA) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Kozlov updated IGNITE-10015:
---
Attachment: hs_err_pid9126.log

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Assignee: Alexey Goncharuk
>Priority: Critical
> Fix For: 2.7
>
> Attachments: hs_err_pid9126.log
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2018-10-26 Thread Vladimir Ozerov (JIRA) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Ozerov updated IGNITE-10015:
-
Ignite Flags:   (was: Docs Required)

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Priority: Critical
> Fix For: 2.7
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problema

[jira] [Updated] (IGNITE-10015) Sporadic JVM crash due to restart nodes

2018-10-26 Thread Sergey Kozlov (JIRA) 


 [ 
https://issues.apache.org/jira/browse/IGNITE-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Kozlov updated IGNITE-10015:
---
Summary: Sporadic JVM  crash due to restart nodes  (was: Sprodic JVM  crash 
due to restart nodes)

> Sporadic JVM  crash due to restart nodes
> 
>
> Key: IGNITE-10015
> URL: https://issues.apache.org/jira/browse/IGNITE-10015
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.7
>Reporter: Sergey Kozlov
>Priority: Critical
> Fix For: 2.7
>
>
> 1. Start 4 node cluster with pre-configured TTL caches.
> 2. Some 4 node may crash:
> {noformat}
> [22:43:01,485][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_002, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_013, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_001, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_012, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_004, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_015, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,005][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_003, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,006][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_014, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=ignite-sys-cache, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_011, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,007][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_010, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_009, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_006, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_005, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_016, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_008, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,008][INFO][sys-#73][GridCacheProcessor] Finish proxy 
> initialization, cacheName=cache_007, 
> localNodeId=e8ad11f2-c6dd-4683-b449-44a726d715fd
> [22:43:02,012][INFO][db-checkpoint-thread-#68][GridCacheDatabaseSharedManager]
>  Checkpoint started [checkpointId=214d43f2-6096-4b42-ab0f-52b7f98078f4, 
> startPtr=FileWALPointer [idx=0, fileOff=513096, len=16483], 
> checkpointLockWait=0ms, checkpointLockHoldTime=23ms, 
> walCpRecordFsyncDuration=880ms, pages=238, reason='timeout']
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGBUS (0x7) at pc=0x7f0aa29d8522, pid=12344, tid=0x7f08b15f5700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 
> 1.8.0_161-b12)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode 
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libzip.so+0x12522]  newEntry+0x62
> #
> # Core dump written. Default location: 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/core or core.12344
> #
> # An error report file with more information is saved as:
> # 
> /var/lib/teamcity/data/work/dd4d79acf76cc870/i2test/suites/hs_err_pid12344.log
> Compiled method (nm)7845  558 n 0   
> java.util.zip.ZipFile::getEntry (native)
>  total in heap  [0x7f0a8d3d1850,0x7f0a8d3d1bc0] = 880
>  relocation [0x7f0a8d3d1978,0x7f0a8d3d19c0] = 72
>  main code  [0x7f0a8d3d19c0,0x7f0a8d3d1bc0] = 512
> [thread 139675315439360 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside