[jira] [Commented] (CASSANDRA-13335) AssertionError in PerDiskMemtableFlushWrite when comparing TimeUUIDType

2017-07-12 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083575#comment-16083575
 ] 

Malte Pickhan commented on CASSANDRA-13335:
---

We are experiencing the same issue on a 3.9 cluster. [~irit.loy.orion] how did 
you work around that?

> AssertionError in PerDiskMemtableFlushWrite when comparing TimeUUIDType
> ---
>
> Key: CASSANDRA-13335
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13335
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.9
> Single DC
> 6 Nodes
> RF 3
> OS: Ubuntu 16.04
>Reporter: Irit Loy
>
> We are running 3.9  cluster with 6 nodes.
> During a long run we occasionally encounter an AssertionError on one or two 
> of the nodes.
> ERROR [PerDiskMemtableFlushWriter_0:4] 2017-03-02 13:34:37,575 
> CassandraDaemon.java:226 - Exception in thread 
> Thread[PerDiskMemtableFlushWriter_0:4,5,main]
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:159) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:121)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:161)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
>  ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
> at 
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380) 
> ~[apache-cassandra-3.9.jar:3.9]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_111]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12839) CommitLogReplayException: Unexpected error deserializing mutation

2017-03-07 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15899527#comment-15899527
 ] 

Malte Pickhan commented on CASSANDRA-12839:
---

[~jjirsa]Sorry for the delayed response. We actually recovered the node by 
replaying from a healthy one.

> CommitLogReplayException: Unexpected error deserializing mutation
> -
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>Priority: Critical
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was not 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13051) SSTable Corruption - Partition Key fails assertion

2016-12-16 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-13051:
--
Description: 
When running a repair the following exception is triggered:

{code}
java.lang.AssertionError: null  
at 
org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) 
~[apache-cassandra-3.7.jar:3.7]  
at 
java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:794)
 ~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546) 
~[na:1.8.0_91]  
at org.apache.cassandra.db.Memtable.getPartition(Memtable.java:355) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionController.maxPurgeableTimestamp(CompactionController.java:221)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator$Purger.getMaxPurgeableTimestamp(CompactionIterator.java:304)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.lambda$new$0(PurgeFunction.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DeletionPurger.shouldPurge(DeletionPurger.java:33) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:386) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:120) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
 ~[apache-cassandra-3.7.jar:3.7]  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_91]  
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]  
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]  
{code}

One thing which would be nice is, to provide an actual message in the assertion 
in order to avoid this "null" string.  Furthermore it would be great to provide 
the data which caused the assertion to fail.

Actually I have no clue, how we triggered this, but I will see if I can find 
out something...

The only similiar Issue I found in the tracker was CASSANDRA-11074.

  was:
When running a repair the following exception is triggered:

{code}
java.lang.AssertionError: null  
at 
org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
 ~[apache-cassandra-3.7.ja

[jira] [Updated] (CASSANDRA-13051) SSTable Corruption - Partition Key fails assertion

2016-12-16 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-13051:
--
Description: 
When running a repair the following exception is triggered:

{code}
java.lang.AssertionError: null  
at 
org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) 
~[apache-cassandra-3.7.jar:3.7]  
at 
java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:794)
 ~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546) 
~[na:1.8.0_91]  
at org.apache.cassandra.db.Memtable.getPartition(Memtable.java:355) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionController.maxPurgeableTimestamp(CompactionController.java:221)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator$Purger.getMaxPurgeableTimestamp(CompactionIterator.java:304)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.lambda$new$0(PurgeFunction.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DeletionPurger.shouldPurge(DeletionPurger.java:33) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:386) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:120) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
 ~[apache-cassandra-3.7.jar:3.7]  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_91]  
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]  
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]  
{code}

One thing which would be nice is, to provide an actual message in the assertion 
in order to avoid this "null" string.  Furthermore it would be great to provide 
the data which caused the assertion to fail.

Actually I have no clue, how we triggered this, but I will see if I can find 
out something...

The only similar Issue I found in the tracker was CASSANDRA-11074, but I am not 
sure if this is really related?

  was:
When running a repair the following exception is triggered:

{code}
java.lang.AssertionError: null  
at 
org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPar

[jira] [Created] (CASSANDRA-13051) SSTable Corruption - Partition Key fails assertion

2016-12-16 Thread Malte Pickhan (JIRA) 
Malte Pickhan created CASSANDRA-13051:
-

 Summary: SSTable Corruption - Partition Key fails assertion
 Key: CASSANDRA-13051
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13051
 Project: Cassandra
  Issue Type: Bug
 Environment: Cassandra 3.7
Single DC
5 Nodes
RF 3
NetworkTopologyStrategy
OS: Ubuntu
Reporter: Malte Pickhan


When running a repair the following exception is triggered:

{code}
java.lang.AssertionError: null  
at 
org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) 
~[apache-cassandra-3.7.jar:3.7]  
at 
java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:794)
 ~[na:1.8.0_91]  
at 
java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546) 
~[na:1.8.0_91]  
at org.apache.cassandra.db.Memtable.getPartition(Memtable.java:355) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionController.maxPurgeableTimestamp(CompactionController.java:221)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator$Purger.getMaxPurgeableTimestamp(CompactionIterator.java:304)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.lambda$new$0(PurgeFunction.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DeletionPurger.shouldPurge(DeletionPurger.java:33) 
~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:386) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:120) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
 ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 ~[apache-cassandra-3.7.jar:3.7]  
at 
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
 ~[apache-cassandra-3.7.jar:3.7]  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_91]  
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_91]  
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]  
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]  
{code}

One thing which would be nice is, to provide an actual message in the assertion 
in order to avoid this "null" string.  Furthermore it would be great to provide 
the data which caused the assertion to fail.

Actually I have no clue, how we triggered this, but I will see if I can find 
out something...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-13051) SSTable Corruption - Partition Key fails assertion

2016-12-16 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-13051:
--
Since Version: 3.7

> SSTable Corruption - Partition Key fails assertion
> --
>
> Key: CASSANDRA-13051
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13051
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.7
> Single DC
> 5 Nodes
> RF 3
> NetworkTopologyStrategy
> OS: Ubuntu
>Reporter: Malte Pickhan
>
> When running a repair the following exception is triggered:
> {code}
> java.lang.AssertionError: null  
> at 
> org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655)
>  ~[na:1.8.0_91]  
> at 
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:794)
>  ~[na:1.8.0_91]  
> at 
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
>  ~[na:1.8.0_91]  
> at org.apache.cassandra.db.Memtable.getPartition(Memtable.java:355) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionController.maxPurgeableTimestamp(CompactionController.java:221)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionIterator$Purger.getMaxPurgeableTimestamp(CompactionIterator.java:304)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.lambda$new$0(PurgeFunction.java:38)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.db.DeletionPurger.shouldPurge(DeletionPurger.java:33) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:386) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:120) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
>  ~[apache-cassandra-3.7.jar:3.7]  
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_91]  
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]  
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  ~[na:1.8.0_91]  
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]  
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]  
> {code}
> One thing which would be nice is, to provide an actual message in the 
> assertion in order to avoid this "null" string.  Furthermore it would be 
> great to provide the data which caused the assertion to fail.
> Actually I have no clue, how we triggered this, but I will see if I can find 
> out something...



--
This message was sent by Atlassian JI

[jira] [Commented] (CASSANDRA-12947) Repair not replicating data

2016-12-16 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754027#comment-15754027
 ] 

Malte Pickhan commented on CASSANDRA-12947:
---

No, we didn't as mentioned in the description we've only run an incremental 
repair.

> Repair not replicating data
> ---
>
> Key: CASSANDRA-12947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12947
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.7
> Single DC
> 7 Nodes
> RF 3
> NetworkTopologyStrategy
> OS: Ubuntu
>Reporter: Malte Pickhan
> Fix For: 3.x
>
>
> We experienced strange behaviour of our C* cluster last week.
> In the logs we've seen multiple requests in the logs where we the values read 
> from Cassandra actually have been 'null'. 
> When running the query on a node, we sometimes received the result and 
> sometimes not.
> When setting the CL to LOCAL_QUORUM everything was fine. So far nothing 
> unusual, probably the dataset wasn't replicated to one of the nodes.
> When turning on the tracing and running the query there was following 
> intersting line:
> {quote}
> Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 |   
> $PUBLIC_IP |126 | 127.0.0.1
>  Digest mismatch: org.apache.cassandra.service.DigestMismatchException: 
> Mismatch for key DecoratedKey(-5887526567589486157, 
> 3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs 
> d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
> {quote}
> This is probably related to CASSANDRA-12090?
> The interesting part is, after that we've run a 'nodetool repair -pr', after 
> that the behaviour was still the same and the data randomly not available, 
> depending on which node was hit.
> Only after running a 'nodetool repair -pr -full' the issue was gone.
> Did we miss something here? The point that's bothering me is that the dataset 
> was not replicated.
> Worth to note is probably that some weeks ago we've hit the bug 
> CASSANDRA-12694 and fixed it by scrubbing some tables.
> Any hints/help are appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-12947) Repair not replicating data

2016-11-22 Thread Malte Pickhan (JIRA) 
Malte Pickhan created CASSANDRA-12947:
-

 Summary: Repair not replicating data
 Key: CASSANDRA-12947
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12947
 Project: Cassandra
  Issue Type: Bug
 Environment: Cassandra 3.7
Single DC
7 Nodes
RF 3
NetworkTopologyStrategy
OS: Ubuntu
Reporter: Malte Pickhan


We experienced strange behaviour of our C* cluster last week.

In the logs we've seen multiple requests in the logs where we the values read 
from Cassandra actually have been 'null'. 
When running the query on a node, we sometimes received the result and 
sometimes not.

When setting the CL to LOCAL_QUORUM everything was fine. So far nothing 
unusual, probably the dataset wasn't replicated to one of the nodes.

When turning on the tracing and running the query there was following 
intersting line:

{quote}
Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 |   
$PUBLIC_IP |126 | 127.0.0.1
 Digest mismatch: org.apache.cassandra.service.DigestMismatchException: 
Mismatch for key DecoratedKey(-5887526567589486157, 
3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs 
d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
{quote}

This is probably related to CASSANDRA-12090?

The interesting part is, after that we've run a 'nodetool repair -pr', after 
that the behaviour was still the same and the data randomly not available, 
depending on which node was hit.

Only after running a 'nodetool repair -pr -full' the issue was gone.

Did we miss something here? The point that's bothering me is that the dataset 
was not replicated.

Worth to note is probably that some weeks ago we've hit the bug CASSANDRA-12694 
and fixed it by scrubbing some tables.

Any hints/help are appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-12839) CommitLogReplayException: Unexpected error deserializing mutation

2016-11-02 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628850#comment-15628850
 ] 

Malte Pickhan edited comment on CASSANDRA-12839 at 11/2/16 12:44 PM:
-

Sorry, I cannot attach it since the file is already removed. No, we didn't 
start it like that. We just dropped the commitlog and bootstrapped a fresh node 
to replace it. After that we went through all nodes of the cluster and drained 
it in order to get rid of potentially corrupted commitlogs.


was (Author: malpi):
Sorry, I cannot attach it since the file is already removed. No, we didn't 
start it like that. We just dropped the commitlog and bootstrapped a fresh node 
to replace it.

> CommitLogReplayException: Unexpected error deserializing mutation
> -
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>Priority: Blocker
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was not 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-12839) CommitLogReplayException: Unexpected error deserializing mutation

2016-11-02 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628850#comment-15628850
 ] 

Malte Pickhan commented on CASSANDRA-12839:
---

Sorry, I cannot attach it since the file is already removed. No, we didn't 
start it like that. We just dropped the commitlog and bootstrapped a fresh node 
to replace it.

> CommitLogReplayException: Unexpected error deserializing mutation
> -
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>Priority: Blocker
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was not 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12839) Replaying commitlog doesn't work

2016-10-26 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-12839:
--
Description: 
We wanted to run a *nodetool repair*  on the affected node. This was not 
possible due to CASSANDRA-12694.

Therefore we wanted to shutdown the node in order to run scrubsstables. After 
this was done, bootstrapping the node was not possible anymore. It was not 
possible since it wasn't able to replay the commitlog. Unfortunately we didn't 
run a drain before.

{quote}
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition
{quote}

Two days before that, we actually did some modification on the table and 
dropped a column.



  was:
We wanted to run a *nodetool repair*  on the affected node. This was not 
possible due to CASSANDRA-12694.

Therefore we wanted to shutdown the node in order to run scrubsstables. After 
this was done, bootstrapping the node was not possible anymore. It was possible 
since it wasn't able to replay the commitlog. Unfortunately we didn't run a 
drain before.

{quote}
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition
{quote}

Two days before that, we actually did some modification on the table and 
dropped a column.




> Replaying commitlog doesn't work
> 
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>Priority: Blocker
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was not 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12839) Replaying commitlog doesn't work

2016-10-26 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-12839:
--
Priority: Blocker  (was: Major)

> Replaying commitlog doesn't work
> 
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>Priority: Blocker
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12839) Replaying commitlog doesn't work

2016-10-26 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-12839:
--
Description: 
We wanted to run a *nodetool repair*  on the affected node. This was not 
possible due to CASSANDRA-12694.

Therefore we wanted to shutdown the node in order to run scrubsstables. After 
this was done, bootstrapping the node was not possible anymore. It was possible 
since it wasn't able to replay the commitlog. Unfortunately we didn't run a 
drain before.

{quote}
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition
{quote}

Two days before that, we actually did some modification on the table and 
dropped a column.



  was:
We wanted to run a *nodetool repair*  on the affected node. This was not 
possible due to CASSANDRA-12694.

Therefore we wanted to shutdown the node in order to run scrubsstables. After 
this was done, bootstrapping the node was not possible anymore. It was possible 
since it wasn't able to replay the commitlog. Unfortunately we didn't run a 
drain before.

{quote}
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition
{quote}




> Replaying commitlog doesn't work
> 
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}
> Two days before that, we actually did some modification on the table and 
> dropped a column.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-12839) Replaying commitlog doesn't work

2016-10-26 Thread Malte Pickhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Malte Pickhan updated CASSANDRA-12839:
--
Environment: Ubuntu, Cassandra 3.7, RF3  (was: Ubuntu, Cassandra 3.7)

> Replaying commitlog doesn't work
> 
>
> Key: CASSANDRA-12839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
> Project: Cassandra
>  Issue Type: Bug
> Environment: Ubuntu, Cassandra 3.7, RF3
>Reporter: Malte Pickhan
>
> We wanted to run a *nodetool repair*  on the affected node. This was not 
> possible due to CASSANDRA-12694.
> Therefore we wanted to shutdown the node in order to run scrubsstables. After 
> this was done, bootstrapping the node was not possible anymore. It was 
> possible since it wasn't able to replay the commitlog. Unfortunately we 
> didn't run a drain before.
> {quote}
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: java.io.IOError: java.io.IOException: Corrupt empty row 
> found in unfiltered partition
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-12839) Replaying commitlog doesn't work

2016-10-26 Thread Malte Pickhan (JIRA) 
Malte Pickhan created CASSANDRA-12839:
-

 Summary: Replaying commitlog doesn't work
 Key: CASSANDRA-12839
 URL: https://issues.apache.org/jira/browse/CASSANDRA-12839
 Project: Cassandra
  Issue Type: Bug
 Environment: Ubuntu, Cassandra 3.7
Reporter: Malte Pickhan


We wanted to run a *nodetool repair*  on the affected node. This was not 
possible due to CASSANDRA-12694.

Therefore we wanted to shutdown the node in order to run scrubsstables. After 
this was done, bootstrapping the node was not possible anymore. It was possible 
since it wasn't able to replay the commitlog. Unfortunately we didn't run a 
drain before.

{quote}
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Unexpected error deserializing mutation; saved to 
/tmp/mutation4616409546670104469dat.  This may be caused by replaying a 
mutation against a table with the same name but incompatible schema.  Exception 
follows: java.io.IOError: java.io.IOException: Corrupt empty row found in 
unfiltered partition
{quote}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8927) Mark libjna-java + libjna-jni as incompatible in debian package

2016-01-21 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110408#comment-15110408
 ] 

Malte Pickhan commented on CASSANDRA-8927:
--

I'm not really sure if it's a good idea to override kernel libraries?

> Mark libjna-java + libjna-jni as incompatible in debian package
> ---
>
> Key: CASSANDRA-8927
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8927
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Debian
>Reporter: Robert Stupp
>Assignee: Michael Shuler
>Priority: Minor
> Fix For: 3.0.x
>
>
> Current Debian (Wheezy) might bring {{libjna-java}} in version 3.2.7-4, which 
> has incompatible {{libjnadispatch.so}} because since C* 2.1 we use JNA 4.0.0 
> (the native stuff changed):
> jna.jar includes all binaries for all supported platforms - so there's no 
> need for libjna installed separately.
> Since CASSANDRA-8714 has been committed, the incompatibility manifests in 
> {{java.lang.NoClassDefFoundError: Could not initialize class 
> com.sun.jna.Native}} (which is caused by outdated libjna-java installed via 
> apt).
> Note: Debian jessie adds new package {{libjna-jni}} (4.1.0-1) in addition to 
> {{libjna-java}} (4.1.0-1) - both contain the {{libjnidispatch.so}}. Although 
> these seem to work, we might hit the same issue when there's a need to 
> upgrade JNA to 4.2.x sometime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8927) Mark libjna-java + libjna-jni as incompatible in debian package

2016-01-14 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098187#comment-15098187
 ] 

Malte Pickhan commented on CASSANDRA-8927:
--

Actually I resolved the issue in my case by just removing 'libjna-java'.

> Mark libjna-java + libjna-jni as incompatible in debian package
> ---
>
> Key: CASSANDRA-8927
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8927
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Debian
>Reporter: Robert Stupp
>Assignee: Michael Shuler
>Priority: Minor
> Fix For: 3.0.x
>
>
> Current Debian (Wheezy) might bring {{libjna-java}} in version 3.2.7-4, which 
> has incompatible {{libjnadispatch.so}} because since C* 2.1 we use JNA 4.0.0 
> (the native stuff changed):
> jna.jar includes all binaries for all supported platforms - so there's no 
> need for libjna installed separately.
> Since CASSANDRA-8714 has been committed, the incompatibility manifests in 
> {{java.lang.NoClassDefFoundError: Could not initialize class 
> com.sun.jna.Native}} (which is caused by outdated libjna-java installed via 
> apt).
> Note: Debian jessie adds new package {{libjna-jni}} (4.1.0-1) in addition to 
> {{libjna-java}} (4.1.0-1) - both contain the {{libjnidispatch.so}}. Although 
> these seem to work, we might hit the same issue when there's a need to 
> upgrade JNA to 4.2.x sometime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8927) Mark libjna-java + libjna-jni as incompatible in debian package

2016-01-14 Thread Malte Pickhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098162#comment-15098162
 ] 

Malte Pickhan commented on CASSANDRA-8927:
--

Actually I currently hit exactly this issue, is there any advice as workaround.

I get the following error during startup:

There is an incompatible JNA native library installed on this system
/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib.
To resolve this issue you may do one of the following:
 - remove or uninstall the offending library
 - set the system property jna.nosys=true
 - set jna.boot.library.path to include the path to the version of the 
   jnidispatch library included with the JNA jar file you are using

As far as i understood adding `jna.nosys=true` to the cassandra-env.sh might 
have the implication that cassandra is unable to allocate more memory.

> Mark libjna-java + libjna-jni as incompatible in debian package
> ---
>
> Key: CASSANDRA-8927
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8927
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Debian
>Reporter: Robert Stupp
>Assignee: Michael Shuler
>Priority: Minor
> Fix For: 3.0.x
>
>
> Current Debian (Wheezy) might bring {{libjna-java}} in version 3.2.7-4, which 
> has incompatible {{libjnadispatch.so}} because since C* 2.1 we use JNA 4.0.0 
> (the native stuff changed):
> jna.jar includes all binaries for all supported platforms - so there's no 
> need for libjna installed separately.
> Since CASSANDRA-8714 has been committed, the incompatibility manifests in 
> {{java.lang.NoClassDefFoundError: Could not initialize class 
> com.sun.jna.Native}} (which is caused by outdated libjna-java installed via 
> apt).
> Note: Debian jessie adds new package {{libjna-jni}} (4.1.0-1) in addition to 
> {{libjna-java}} (4.1.0-1) - both contain the {{libjnidispatch.so}}. Although 
> these seem to work, we might hit the same issue when there's a need to 
> upgrade JNA to 4.2.x sometime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)