[jira] [Created] (CASSANDRA-6366) Corrupt SSTables

2013-11-17 Thread Matt Jurik (JIRA) 
Matt Jurik created CASSANDRA-6366:
-

 Summary: Corrupt SSTables
 Key: CASSANDRA-6366
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
 Project: Cassandra
  Issue Type: Bug
 Environment: 1.2.10
Reporter: Matt Jurik


We ran into some corrupt sstables on one of our 8-node clusters running 1.2.10 
(since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a single 
node. After doing a "nodetool scrub" and then a "nodetool -pr repair" for the 
cluster, we were left with 2 nodes reporting 3 corrupt sstables.

All nodes appear healthy; fsck and our raid controllers report no issues. The 
sstables were written out during normal operation; there were no machine 
restarts or failures anywhere near the sstable file timestamps.

Curiously, I figured out how to read all 3 of our corrupt sstables, though I 
have no idea why this works. Additionally, it seems that I'm able to read all 
OnDiskAtoms as specified in the row header, so the data seems intact.

{code}
diff --git 
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java 
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 381fdb9..8fce5f7 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements 
Comparable dataSize
+ */
 return inputWithTracker.getBytesRead() < dataSize;
 }
 
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java 
b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
index 1df5842..718324c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
@@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
 {
 try
 {
-if (row != null)
-dfile.seek(finishedAt);
+// Magically read corrupt sstables...
+// if (row != null)
+// dfile.seek(finishedAt);
 assert !dfile.isEOF();
 
 // Read data header
diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
b/src/java/org/apache/cassandra/tools/SSTableExport.java
index 05fe9f6..ed61010 100644
--- a/src/java/org/apache/cassandra/tools/SSTableExport.java
+++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
@@ -432,7 +432,7 @@ public class SSTableExport
  */
 public static void export(Descriptor desc, String[] excludes) throws 
IOException
 {
-export(desc, System.out, excludes);
+export(desc, new PrintStream("json"), excludes);
 }
 
 /**
{code}

Otherwise, I get a stacktrace such as:

{code}
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
dataSize of 72339146324312065 starting at 80476328 would be larger than file 
/Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
 length 109073657
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:176)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:84)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:70)
at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:422)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:435)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.io.IOException: dataSize of 72339146324312065 starting at 
80476328 would be larger than file 
/Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
 length 109073657
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:132)
... 14 more
{code}

Any help on the matter is appreciated.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (CASSANDRA-6366) Corrupt SSTables

2013-11-17 Thread Matt Jurik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Jurik updated CASSANDRA-6366:
--

Environment: 1.2.10, LeveledCompaction, CQL3  (was: 1.2.10)

> Corrupt SSTables
> 
>
> Key: CASSANDRA-6366
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
> Project: Cassandra
>  Issue Type: Bug
> Environment: 1.2.10, LeveledCompaction, CQL3
>Reporter: Matt Jurik
>
> We ran into some corrupt sstables on one of our 8-node clusters running 
> 1.2.10 (since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a 
> single node. After doing a "nodetool scrub" and then a "nodetool -pr repair" 
> for the cluster, we were left with 2 nodes reporting 3 corrupt sstables.
> All nodes appear healthy; fsck and our raid controllers report no issues. The 
> sstables were written out during normal operation; there were no machine 
> restarts or failures anywhere near the sstable file timestamps.
> Curiously, I figured out how to read all 3 of our corrupt sstables, though I 
> have no idea why this works. Additionally, it seems that I'm able to read all 
> OnDiskAtoms as specified in the row header, so the data seems intact.
> {code}
> diff --git 
> a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> index 381fdb9..8fce5f7 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> @@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements 
> Comparable  
>  public boolean hasNext()
>  {
> + /*
> + * For each row where corruption is reported, it is the case that we 
> read more data from the preceeding row
> + * than specified by dataSize. That is, this iterator will terminate 
> with:
> + * inputWithTracker.getBytesRead() > dataSize
> + */
>  return inputWithTracker.getBytesRead() < dataSize;
>  }
>  
> diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> index 1df5842..718324c 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> @@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
>  {
>  try
>  {
> -if (row != null)
> -dfile.seek(finishedAt);
> +// Magically read corrupt sstables...
> +// if (row != null)
> +// dfile.seek(finishedAt);
>  assert !dfile.isEOF();
>  
>  // Read data header
> diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
> b/src/java/org/apache/cassandra/tools/SSTableExport.java
> index 05fe9f6..ed61010 100644
> --- a/src/java/org/apache/cassandra/tools/SSTableExport.java
> +++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
> @@ -432,7 +432,7 @@ public class SSTableExport
>   */
>  public static void export(Descriptor desc, String[] excludes) throws 
> IOException
>  {
> -export(desc, System.out, excludes);
> +export(desc, new PrintStream("json"), excludes);
>  }
>  
>  /**
> {code}
> Otherwise, I get a stacktrace such as:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
> dataSize of 72339146324312065 starting at 80476328 would be larger than file 
> /Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
>  length 109073657
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:176)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:84)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:70)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:422)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:435)
> at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.ref

[jira] [Updated] (CASSANDRA-6366) Corrupt SSTables

2013-11-17 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-6366:
--

Priority: Minor  (was: Major)

Scrub should be able to fix a corrupt row size like this.  What does it log 
when you scrub it?

> Corrupt SSTables
> 
>
> Key: CASSANDRA-6366
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
> Project: Cassandra
>  Issue Type: Bug
> Environment: 1.2.10, LeveledCompaction, CQL3
>Reporter: Matt Jurik
>Priority: Minor
>
> We ran into some corrupt sstables on one of our 8-node clusters running 
> 1.2.10 (since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a 
> single node. After doing a "nodetool scrub" and then a "nodetool -pr repair" 
> for the cluster, we were left with 2 nodes reporting 3 corrupt sstables.
> All nodes appear healthy; fsck and our raid controllers report no issues. The 
> sstables were written out during normal operation; there were no machine 
> restarts or failures anywhere near the sstable file timestamps.
> Curiously, I figured out how to read all 3 of our corrupt sstables, though I 
> have no idea why this works. Additionally, it seems that I'm able to read all 
> OnDiskAtoms as specified in the row header, so the data seems intact.
> {code}
> diff --git 
> a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> index 381fdb9..8fce5f7 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> @@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements 
> Comparable  
>  public boolean hasNext()
>  {
> + /*
> + * For each row where corruption is reported, it is the case that we 
> read more data from the preceeding row
> + * than specified by dataSize. That is, this iterator will terminate 
> with:
> + * inputWithTracker.getBytesRead() > dataSize
> + */
>  return inputWithTracker.getBytesRead() < dataSize;
>  }
>  
> diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> index 1df5842..718324c 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> @@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
>  {
>  try
>  {
> -if (row != null)
> -dfile.seek(finishedAt);
> +// Magically read corrupt sstables...
> +// if (row != null)
> +// dfile.seek(finishedAt);
>  assert !dfile.isEOF();
>  
>  // Read data header
> diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
> b/src/java/org/apache/cassandra/tools/SSTableExport.java
> index 05fe9f6..ed61010 100644
> --- a/src/java/org/apache/cassandra/tools/SSTableExport.java
> +++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
> @@ -432,7 +432,7 @@ public class SSTableExport
>   */
>  public static void export(Descriptor desc, String[] excludes) throws 
> IOException
>  {
> -export(desc, System.out, excludes);
> +export(desc, new PrintStream("json"), excludes);
>  }
>  
>  /**
> {code}
> Otherwise, I get a stacktrace such as:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
> dataSize of 72339146324312065 starting at 80476328 would be larger than file 
> /Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
>  length 109073657
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:176)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:84)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:70)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:422)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:435)
> at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:517)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>

[jira] [Created] (CASSANDRA-6367) Enable purge of local hints

2013-11-17 Thread Cyril Scetbon (JIRA) 
Cyril Scetbon created CASSANDRA-6367:


 Summary: Enable purge of local hints
 Key: CASSANDRA-6367
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6367
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Cyril Scetbon


We should have a new nodetool command (purgelocalhints as a suggestion) to 
locally truncate the system.hints cf and not on all nodes as it currently works 
if we use truncate DDL command. We could have access to this new functionality 
through JMX too

see thread 
http://mail-archives.apache.org/mod_mbox/cassandra-dev/201311.mbox/%3c8e5f2112-8d98-4f6b-aa49-08ba3ff00...@free.fr%3e



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (CASSANDRA-6367) Enable purge of local hints

2013-11-17 Thread Cyril Scetbon (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cyril Scetbon updated CASSANDRA-6367:
-

Description: 
We should have a new nodetool command (purgelocalhints as a suggestion) to 
locally truncate the system.hints cf and not on all nodes as it currently works 
if we use the TRUNCATE DDL command. We could have access to this new 
functionality through JMX too

see thread 
http://mail-archives.apache.org/mod_mbox/cassandra-dev/201311.mbox/%3c8e5f2112-8d98-4f6b-aa49-08ba3ff00...@free.fr%3e

  was:
We should have a new nodetool command (purgelocalhints as a suggestion) to 
locally truncate the system.hints cf and not on all nodes as it currently works 
if we use truncate DDL command. We could have access to this new functionality 
through JMX too

see thread 
http://mail-archives.apache.org/mod_mbox/cassandra-dev/201311.mbox/%3c8e5f2112-8d98-4f6b-aa49-08ba3ff00...@free.fr%3e


> Enable purge of local hints
> ---
>
> Key: CASSANDRA-6367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6367
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Core
>Reporter: Cyril Scetbon
>
> We should have a new nodetool command (purgelocalhints as a suggestion) to 
> locally truncate the system.hints cf and not on all nodes as it currently 
> works if we use the TRUNCATE DDL command. We could have access to this new 
> functionality through JMX too
> see thread 
> http://mail-archives.apache.org/mod_mbox/cassandra-dev/201311.mbox/%3c8e5f2112-8d98-4f6b-aa49-08ba3ff00...@free.fr%3e



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-6366) Corrupt SSTables

2013-11-17 Thread Matt Jurik (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824975#comment-13824975
 ] 

Matt Jurik commented on CASSANDRA-6366:
---

Ah, I guess scrub did fix this; I've found the scrub log lines for each corrupt 
sstable and it seems that scrub wrote out the same number of rows that I was 
able to read with my patch. Does this mean that data integrity is ok? Why did 
this occur?

Leveling seems to break when there are corrupt sstables; L0 explodes in size.

> Corrupt SSTables
> 
>
> Key: CASSANDRA-6366
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
> Project: Cassandra
>  Issue Type: Bug
> Environment: 1.2.10, LeveledCompaction, CQL3
>Reporter: Matt Jurik
>Priority: Minor
>
> We ran into some corrupt sstables on one of our 8-node clusters running 
> 1.2.10 (since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a 
> single node. After doing a "nodetool scrub" and then a "nodetool -pr repair" 
> for the cluster, we were left with 2 nodes reporting 3 corrupt sstables.
> All nodes appear healthy; fsck and our raid controllers report no issues. The 
> sstables were written out during normal operation; there were no machine 
> restarts or failures anywhere near the sstable file timestamps.
> Curiously, I figured out how to read all 3 of our corrupt sstables, though I 
> have no idea why this works. Additionally, it seems that I'm able to read all 
> OnDiskAtoms as specified in the row header, so the data seems intact.
> {code}
> diff --git 
> a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> index 381fdb9..8fce5f7 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> @@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements 
> Comparable  
>  public boolean hasNext()
>  {
> + /*
> + * For each row where corruption is reported, it is the case that we 
> read more data from the preceeding row
> + * than specified by dataSize. That is, this iterator will terminate 
> with:
> + * inputWithTracker.getBytesRead() > dataSize
> + */
>  return inputWithTracker.getBytesRead() < dataSize;
>  }
>  
> diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java 
> b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> index 1df5842..718324c 100644
> --- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> +++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
> @@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
>  {
>  try
>  {
> -if (row != null)
> -dfile.seek(finishedAt);
> +// Magically read corrupt sstables...
> +// if (row != null)
> +// dfile.seek(finishedAt);
>  assert !dfile.isEOF();
>  
>  // Read data header
> diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java 
> b/src/java/org/apache/cassandra/tools/SSTableExport.java
> index 05fe9f6..ed61010 100644
> --- a/src/java/org/apache/cassandra/tools/SSTableExport.java
> +++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
> @@ -432,7 +432,7 @@ public class SSTableExport
>   */
>  public static void export(Descriptor desc, String[] excludes) throws 
> IOException
>  {
> -export(desc, System.out, excludes);
> +export(desc, new PrintStream("json"), excludes);
>  }
>  
>  /**
> {code}
> Otherwise, I get a stacktrace such as:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: 
> dataSize of 72339146324312065 starting at 80476328 would be larger than file 
> /Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
>  length 109073657
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:176)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:84)
> at 
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:70)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
> at 
> org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
> at org.apache.cassandra.tools.SSTableExport.export(SSTableExpo

[jira] [Created] (CASSANDRA-6368) java.lang.NumberFormatException: For input string: "140804036566258204771707954633792970268"

2013-11-17 Thread *$^¨%`£ 
Olivier Lamy (*$^¨%`£) created CASSANDRA-6368:
-

 Summary: java.lang.NumberFormatException: For input string: 
"140804036566258204771707954633792970268"
 Key: CASSANDRA-6368
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6368
 Project: Cassandra
  Issue Type: Bug
Reporter: Olivier Lamy (*$^¨%`£)


ERROR 13:24:31,790 Error occurred during processing of message.
java.lang.NumberFormatException: For input string: 
"140804036566258204771707954633792970268"
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:422)
at java.lang.Long.valueOf(Long.java:525)
at 
org.apache.cassandra.dht.Murmur3Partitioner$1.fromString(Murmur3Partitioner.java:188)
at 
org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:936)
at 
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3454)
at 
org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3442)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-6368) java.lang.NumberFormatException: For input string: "140804036566258204771707954633792970268"

2013-11-17 Thread *$^¨%`£ 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825042#comment-13825042
 ] 

Olivier Lamy (*$^¨%`£) commented on CASSANDRA-6368:
---

using cassandra 1.2.5

mb-olamy:cassandra-maven-plugin-1.x olamy$ cassandra -f
xss =  -ea -javaagent:/usr/local/Cellar/cassandra/1.2.5/jamm-0.2.5.jar 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms2048M -Xmx2048M 
-Xmn512M -XX:+HeapDumpOnOutOfMemoryError
 INFO 13:26:04,270 Logging initialized
 INFO 13:26:04,281 JVM vendor/version: Java HotSpot(TM) 64-Bit Server 
VM/1.6.0_65
 INFO 13:26:04,281 Heap size: 2093809664/2093809664
 INFO 13:26:04,281 Classpath: 
/usr/local/etc/cassandra:/usr/local/Cellar/cassandra/1.2.5/build/classes/main:/usr/local/Cellar/cassandra/1.2.5/build/classes/thrift:/usr/local/Cellar/cassandra/1.2.5/antlr-3.2.jar:/usr/local/Cellar/cassandra/1.2.5/apache-cassandra-1.2.5.jar:/usr/local/Cellar/cassandra/1.2.5/apache-cassandra-clientutil-1.2.5.jar:/usr/local/Cellar/cassandra/1.2.5/apache-cassandra-thrift-1.2.5.jar:/usr/local/Cellar/cassandra/1.2.5/avro-1.4.0-fixes.jar:/usr/local/Cellar/cassandra/1.2.5/avro-1.4.0-sources-fixes.jar:/usr/local/Cellar/cassandra/1.2.5/commons-cli-1.1.jar:/usr/local/Cellar/cassandra/1.2.5/commons-codec-1.2.jar:/usr/local/Cellar/cassandra/1.2.5/commons-lang-2.6.jar:/usr/local/Cellar/cassandra/1.2.5/compress-lzf-0.8.4.jar:/usr/local/Cellar/cassandra/1.2.5/concurrentlinkedhashmap-lru-1.3.jar:/usr/local/Cellar/cassandra/1.2.5/guava-13.0.1.jar:/usr/local/Cellar/cassandra/1.2.5/high-scale-lib-1.1.2.jar:/usr/local/Cellar/cassandra/1.2.5/jackson-core-asl-1.9.2.jar:/usr/local/Cellar/cassandra/1.2.5/jackson-mapper-asl-1.9.2.jar:/usr/local/Cellar/cassandra/1.2.5/jamm-0.2.5.jar:/usr/local/Cellar/cassandra/1.2.5/jbcrypt-0.3m.jar:/usr/local/Cellar/cassandra/1.2.5/jline-1.0.jar:/usr/local/Cellar/cassandra/1.2.5/json-simple-1.1.jar:/usr/local/Cellar/cassandra/1.2.5/libthrift-0.7.0.jar:/usr/local/Cellar/cassandra/1.2.5/log4j-1.2.16.jar:/usr/local/Cellar/cassandra/1.2.5/lz4-1.1.0.jar:/usr/local/Cellar/cassandra/1.2.5/metrics-core-2.0.3.jar:/usr/local/Cellar/cassandra/1.2.5/netty-3.5.9.Final.jar:/usr/local/Cellar/cassandra/1.2.5/servlet-api-2.5-20081211.jar:/usr/local/Cellar/cassandra/1.2.5/slf4j-api-1.7.2.jar:/usr/local/Cellar/cassandra/1.2.5/slf4j-log4j12-1.7.2.jar:/usr/local/Cellar/cassandra/1.2.5/snakeyaml-1.6.jar:/usr/local/Cellar/cassandra/1.2.5/snappy-java-1.0.4.1.jar:/usr/local/Cellar/cassandra/1.2.5/snaptree-0.1.jar:/usr/local/Cellar/cassandra/1.2.5/jamm-0.2.5.jar
 INFO 13:26:04,283 JNA not found. Native methods will be disabled.
 INFO 13:26:04,294 Loading settings from 
file:/usr/local/etc/cassandra/cassandra.yaml
 INFO 13:26:04,526 Data files directories: [/usr/local/var/lib/cassandra/data]
 INFO 13:26:04,526 Commit log directory: /usr/local/var/lib/cassandra/commitlog
 INFO 13:26:04,526 DiskAccessMode 'auto' determined to be mmap, indexAccessMode 
is mmap
 INFO 13:26:04,527 disk_failure_policy is stop
 INFO 13:26:04,530 Global memtable threshold is enabled at 665MB
 INFO 13:26:04,619 Not using multi-threaded compaction
 INFO 13:26:04,808 Initializing key cache with capacity of 99 MBs.
 INFO 13:26:04,814 Scheduling key cache save to each 14400 seconds (going to 
save all keys).
 INFO 13:26:04,815 Initializing row cache with capacity of 0 MBs and provider 
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 13:26:04,819 Scheduling row cache save to each 0 seconds (going to save 
all keys).


> java.lang.NumberFormatException: For input string: 
> "140804036566258204771707954633792970268"
> 
>
> Key: CASSANDRA-6368
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6368
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Olivier Lamy (*$^¨%`£)
>
> ERROR 13:24:31,790 Error occurred during processing of message.
> java.lang.NumberFormatException: For input string: 
> "140804036566258204771707954633792970268"
>   at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
>   at java.lang.Long.parseLong(Long.java:422)
>   at java.lang.Long.valueOf(Long.java:525)
>   at 
> org.apache.cassandra.dht.Murmur3Partitioner$1.fromString(Murmur3Partitioner.java:188)
>   at 
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:936)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3454)
>   at 
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3442)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServe

[jira] [Updated] (CASSANDRA-6345) Endpoint cache invalidation causes CPU spike (on vnode rings?)

2013-11-17 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-6345:
--

Attachment: 6345-v3.txt

I think we can craft a simpler solution (v3) by using an AtomicReference to the 
TM clone.  This removes the possibility of deadlock since clearEndpointCache 
now only makes non-blocking calls.

I've also refined it to use a Striped per-keyToken, as well as 
synchronizing the TM clone itself, since concurrent endpoint computation is 
fine.

> Endpoint cache invalidation causes CPU spike (on vnode rings?)
> --
>
> Key: CASSANDRA-6345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6345
> Project: Cassandra
>  Issue Type: Bug
> Environment: 30 nodes total, 2 DCs
> Cassandra 1.2.11
> vnodes enabled (256 per node)
>Reporter: Rick Branson
>Assignee: Jonathan Ellis
> Fix For: 1.2.12, 2.0.3
>
> Attachments: 6345-rbranson-v2.txt, 6345-rbranson.txt, 6345-v2.txt, 
> 6345-v3.txt, 6345.txt, half-way-thru-6345-rbranson-patch-applied.png
>
>
> We've observed that events which cause invalidation of the endpoint cache 
> (update keyspace, add/remove nodes, etc) in AbstractReplicationStrategy 
> result in several seconds of thundering herd behavior on the entire cluster. 
> A thread dump shows over a hundred threads (I stopped counting at that point) 
> with a backtrace like this:
> at java.net.Inet4Address.getAddress(Inet4Address.java:288)
> at 
> org.apache.cassandra.locator.TokenMetadata$1.compare(TokenMetadata.java:106)
> at 
> org.apache.cassandra.locator.TokenMetadata$1.compare(TokenMetadata.java:103)
> at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:351)
> at java.util.TreeMap.getEntry(TreeMap.java:322)
> at java.util.TreeMap.get(TreeMap.java:255)
> at 
> com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:200)
> at 
> com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:117)
> at com.google.common.collect.TreeMultimap.put(TreeMultimap.java:74)
> at 
> com.google.common.collect.AbstractMultimap.putAll(AbstractMultimap.java:273)
> at com.google.common.collect.TreeMultimap.putAll(TreeMultimap.java:74)
> at 
> org.apache.cassandra.utils.SortedBiMultiValMap.create(SortedBiMultiValMap.java:60)
> at 
> org.apache.cassandra.locator.TokenMetadata.cloneOnlyTokenMap(TokenMetadata.java:598)
> at 
> org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:104)
> at 
> org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2671)
> at 
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:375)
> It looks like there's a large amount of cost in the 
> TokenMetadata.cloneOnlyTokenMap that 
> AbstractReplicationStrategy.getNaturalEndpoints is calling each time there is 
> a cache miss for an endpoint. It seems as if this would only impact clusters 
> with large numbers of tokens, so it's probably a vnodes-only issue.
> Proposal: In AbstractReplicationStrategy.getNaturalEndpoints(), cache the 
> cloned TokenMetadata instance returned by TokenMetadata.cloneOnlyTokenMap(), 
> wrapping it with a lock to prevent stampedes, and clearing it in 
> clearEndpointCache(). Thoughts?



--
This message was sent by Atlassian JIRA
(v6.1#6144)