[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163419#comment-17163419
]
Romain Anselin commented on CASSANDRA-14813:
Just an FYI we encountered this issue on JDK 8_241. The issue was due to faulty
disk sector so worth checking your syslog and run a SMART diag on the node if
this occur
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Normal
> Labels: StubRoutines, crash, updateBytesCRC32
> Attachments: f3.png, f4.png, hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by CASSANDRA-14283 in our
> production system, which runs on Cassandra 3.11.2. We noticed that this issue
> has been resolved in Cassandra 3.11.3 (CASSANDRA-14284), thus we upgrade our
> system to 3.11.3. However, this induce more frequent crash，as shown in the
> following screenshots, and the reduced hs file is posted here as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647583#comment-16647583
]
Jinchao Zhang commented on CASSANDRA-14813:
---
You thought this error has something to do with the consumized secondary index,
right? It's a bit of difficult to verify that on this online production system,
but I wiil try.
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Major
> Attachments: f3.png, f4.png, hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by CASSANDRA-14283 in our
> production system, which runs on Cassandra 3.11.2. We noticed that this issue
> has been resolved in Cassandra 3.11.3 (CASSANDRA-14284), thus we upgrade our
> system to 3.11.3. However, this induce more frequent crash，as shown in the
> following screenshots, and the reduced hs file is posted here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647572#comment-16647572
]
Stefan Podkowinski commented on CASSANDRA-14813:
Is this also happening for writes without your custom indexer at work?
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Major
> Attachments: f3.png, f4.png, hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by CASSANDRA-14283 in our
> production system, which runs on Cassandra 3.11.2. We noticed that this issue
> has been resolved in Cassandra 3.11.3 (CASSANDRA-14284), thus we upgrade our
> system to 3.11.3. However, this induce more frequent crash，as shown in the
> following screenshots, and the reduced hs file is posted here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647308#comment-16647308
]
Jinchao Zhang commented on CASSANDRA-14813:
---
Thanks Stefan, we upgrade jvm to the latest JVM8 version, java 8.0_181-b13. but
the same error still happens . It seems that a jvm bug induce this error.
!f4.png!
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Major
> Attachments: hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by
> [CASSANDRA-14283|https://issues.apache.org/jira/browse/CASSANDRA-14283] in
> our production system, which runs on Cassandra 3.11.2. We noticed that this
> issue has been resolved in Cassandra 3.11.3
> ([CASSANDRA-14284|https://issues.apache.org/jira/browse/CASSANDRA-14284]),
> thus we upgrade our system to 3.11.3. However, this induce more frequent
> crash，as show in the following screenshots, and the reduced hs file is posted
> here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646274#comment-16646274
]
Stefan Podkowinski commented on CASSANDRA-14813:
JVM 8.0_101 is really old, please try to update to see if this can be
reproduced in a more recent JVM version.
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Major
> Attachments: hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by
> [CASSANDRA-14283|https://issues.apache.org/jira/browse/CASSANDRA-14283] in
> our production system, which runs on Cassandra 3.11.2. We noticed that this
> issue has been resolved in Cassandra 3.11.3
> ([CASSANDRA-14284|https://issues.apache.org/jira/browse/CASSANDRA-14284]),
> thus we upgrade our system to 3.11.3. However, this induce more frequent
> crash，as show in the following screenshots, and the reduced hs file is posted
> here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
