[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028086#comment-16028086
]
Ihor Prokopov commented on CASSANDRA-13556:
---
Hi Jeff, thanks for your response a lot!
Here is stacktrace:
{code}
ERROR [CompactionExecutor:2116] 2017-05-26 13:10:00,784
CassandraDaemon.java:226 - Exception in thread
Thread[CompactionExecutor:2116,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/cassandra/apache-cassandra-3.9/data/data/toweya/fetcher-8a86a1b0f4f611e6a6db373124acc314/mc-178837-big-Data.db
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:112)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:500)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:360)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:106)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:141)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:189)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[apache-cassandra-3.9.jar:3.9]
{code}
Also we noticed, that some rows returns without any errors, but fealds' values
were shuffled (I mean that all fealds contains values from other fields).
Best regards, Ihor
> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
>Priority: Critical
> Fix For: 3.9
>
>
> After 3 month of working, we noticed that number of compaction tasks were
> growing (~600 pending tasks). SStables verification shows that some of them
> were corrupted. Repairing didn't help (it was crashing with error).
> Also some of requests (f.e. select * from fetcher where
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next
> error:
> {color:red}
> Traceback (most recent call last):
> File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in
> perform_simple_statement
>
[jira] [Created] (CASSANDRA-13556) Corrupted SSTables
Ihor Prokopov created CASSANDRA-13556:
-
Summary: Corrupted SSTables
Key: CASSANDRA-13556
URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
Project: Cassandra
Issue Type: Bug
Components: Compaction
Environment: CentOS Linux release 7.3.1611 (Core)
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
Python cassandra (DataStax) driver v3.6.0
Reporter: Ihor Prokopov
Fix For: 3.9
After 3 month of working, we noticed that number of compaction tasks were
growing (~600 pending tasks). SStables verification shows that some of them
were corrupted. Repairing didn't help (it was crashing with error).
Also some of requests (f.e. select * from fetcher where
domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next
error:
{color:red}
Traceback (most recent call last):
File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in
perform_simple_statement
result = future.result()
File
"/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
line 3650, in result
raise self._final_exception
error: unpack requires a string argument of length 4
{color}
Table chema:
{quote}
CREATE TABLE fetcher (
domain bigint,
uri bigint,
date date,
content_length int,
elapsed float,
encoding text,
fetched_time bigint,
flinks frozen,
flinks_count int,
html_fingerprint bigint,
indexed boolean,
adult boolean,
kws_count int,
lang_id int,
last_updated bigint,
redirect_url tuple,
revisit_date date,
revisit_interval int,
status_code int,
tokens_fingerprint bigint,
uris frozen
