[jira] [Created] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Terje Marthinussen (JIRA) 
ConcurrentModificationException during node recovery


 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen


Testing some node recovery operations.

In this case:
1. Data is being added/updated as it would in production
2. repair is running on other nodes in the cluster
3. we wiped data on this node and started up again, but before repair was 
actually started on this node (but it had gotten data through the regular data 
feed) we got this error.

I see no indication in the logs that outgoing streams has been started, but the 
node have finished one incoming stream before this (I guess from some other 
node doing repair).

 INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 
155) Opening /data/cassandra/node1/data/JP/test-g-8
 INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 
155) Opening /data/cassandra/node1/data/JP/test-g-10
 INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 
302) Started hinted handoff for endpoint /1.10.42.216
 INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 
358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
 INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
113) Fatal exception in thread Thread[Thread-84,5,main]
java.util.ConcurrentModificationException
at 
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at 
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
113) Fatal exception in thread Thread[Thread-79,5,main]
java.util.ConcurrentModificationException
at 
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at 
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
113) Fatal exception in thread Thread[Thread-83,5,main]
java.util.ConcurrentModificationException
at 
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at 
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
113) Fatal exception in thread Thread[Thread-85,5,main]
java.util.ConcurrentModificationException
at 
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at 
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at 
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction

2011-06-14 Thread Benjamin Coverston (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Coverston updated CASSANDRA-1608:
--

Attachment: 0001-leveldb-style-compaction.patch

Adding a patch for leveldb style compaction. I see this as a 'good start' and 
I'm looking for some further input. I'm not going to be able to work on this 
for the next week or so so I'm putting it here to start some discussion on this 
approach.

This implementation requires no durable manifest.

Ranges are created at SSTable creation (flush or compaction) or sstable index 
creation.

Exponent used for levels is 10.

Preliminary runs show that high write rates do make level 0 to level 1 
promotions back up substantially, but when cleared promotions out of level one 
seem to be very fast.

I found the best performance by removing the compaction throughput throttling 
and setting concurrent compactors to 1.

The SSTable size in this implementation is determined by the flush size in mb 
setting.

The recovery path reads the list of SSTables, groups them by non-overlapping 
ranges then places each range in its appropriate level.

Finally credit is due to the leveldb team as this design was inspired by the 
leveldb implementation.

 Redesigned Compaction
 -

 Key: CASSANDRA-1608
 URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Chris Goffinet
 Attachments: 0001-leveldb-style-compaction.patch


 After seeing the I/O issues in CASSANDRA-1470, I've been doing some more 
 thinking on this subject that I wanted to lay out.
 I propose we redo the concept of how compaction works in Cassandra. At the 
 moment, compaction is kicked off based on a write access pattern, not read 
 access pattern. In most cases, you want the opposite. You want to be able to 
 track how well each SSTable is performing in the system. If we were to keep 
 statistics in-memory of each SSTable, prioritize them based on most accessed, 
 and bloom filter hit/miss ratios, we could intelligently group sstables that 
 are being read most often and schedule them for compaction. We could also 
 schedule lower priority maintenance on SSTable's not often accessed.
 I also propose we limit the size of each SSTable to a fix sized, that gives 
 us the ability to  better utilize our bloom filters in a predictable manner. 
 At the moment after a certain size, the bloom filters become less reliable. 
 This would also allow us to group data most accessed. Currently the size of 
 an SSTable can grow to a point where large portions of the data might not 
 actually be accessed as often.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2759) Scrub could lose increments and replicate that loss

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2759:


Attachment: 0001-Don-t-skip-rows-on-scrub-for-counter-CFs.patch

Attaching patch to simply re-throw the exception instead of skipping the row 
for counter column families.

bq. Only if you actually did have a counter in the column_metadata, right?

right.

 Scrub could lose increments and replicate that loss
 ---

 Key: CASSANDRA-2759
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2759
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: counters
 Fix For: 0.8.1

 Attachments: 0001-Don-t-skip-rows-on-scrub-for-counter-CFs.patch, 
 0001-Renew-nodeId-in-scrub-when-skipping-rows.patch


 If scrub cannot 'repair' a corrupted row, it will skip it. On node A, if the 
 row contains some sub-count for A id, those will be lost forever since A is 
 the source of truth on it's current id. We should thus renew node A id when 
 that happens to avoid this (not unlike we do in cleanup).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2641) AbstractBounds.normalize should deal with overlapping ranges

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049042#comment-13049042
 ] 

Sylvain Lebresne commented on CASSANDRA-2641:
-

bq. it overlaps quite a bit with StorageProxy.getRestrictedRanges: is there 
anything there that can be reused?

getRestrictedRanges splits a range at different tokens. This patch is about 
merging overlapping range as part of normalize. Not sure I follow what could be 
reused here. And in any, I'm in favor of not refactoring anything that is not 
necessary for this patch. This is not worth it.

 AbstractBounds.normalize should deal with overlapping ranges
 

 Key: CASSANDRA-2641
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2641
 Project: Cassandra
  Issue Type: Test
  Components: Core
Reporter: Stu Hood
Assignee: Stu Hood
Priority: Minor
 Fix For: 0.8.1

 Attachments: 0001-Assert-non-overlapping-ranges-in-normalize.txt, 
 0001-Make-normalize-deoverlap-ranges.patch, 
 0002-Don-t-use-overlapping-ranges-in-tests.txt


 Apparently no consumers have encountered it in production, but 
 AbstractBounds.normalize does not handle overlapping ranges. If given 
 overlapping ranges, the output will be sorted but still overlapping, for 
 which SSTableReader.getPositionsForRanges will choose ranges in an SSTable 
 that may overlap.
 We should either add an assert in normalize(), or in getPositionsForRanges() 
 to ensure that this never bites us in production.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135399 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableWriter.java

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 07:55:43 2011
New Revision: 1135399

URL: http://svn.apache.org/viewvc?rev=1135399view=rev
Log:
Seek back after deserializing a row to update cache with
patch by jbellis; reviewed by slebresne for CASSANDRA-2752

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1135399r1=1135398r2=1135399view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jun 14 07:55:43 2011
@@ -51,6 +51,7 @@
  * remove active-pull schema requests (CASSANDRA-2715)
  * avoid marking entire list of sstables as actively being compacted
in multithreaded compaction (CASSANDRA-2765)
+ * seek back after deserializing a row to update cache with (CASSANDRA-2752)
 
 
 0.8.0-final

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java?rev=1135399r1=1135398r2=1135399view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
 Tue Jun 14 07:55:43 2011
@@ -410,8 +410,10 @@ public class SSTableWriter extends SSTab
 if (row == null)
 {
 // If not provided, read from disk.
+long position = dfile.getFilePointer();
 cf = ColumnFamily.create(cfs.metadata);
 
ColumnFamily.serializer().deserializeColumns(dfile, cf, true, true);
+dfile.seek(position);
 }
 else
 {

[jira] [Resolved] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne resolved CASSANDRA-2752.
-

Resolution: Fixed
  Reviewer: slebresne
  Assignee: Jonathan Ellis  (was: Terje Marthinussen)

 repair fails with java.io.EOFException
 --

 Key: CASSANDRA-2752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
Priority: Critical
 Fix For: 0.8.1

 Attachments: 2752.txt


 Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
 INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
 (line 234) Queueing comparison #Differencer #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
 .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
 of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 485) Performing streaming repair of 2 ranges for #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
 fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
 Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
 sections=1 progress=0/36592 - 0%], 1 sstables.
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
 (line 174) Streaming to /1.10.42.82
 ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:9,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 On .82
 ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:12,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-132,5,main]
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.io.EOFException
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049045#comment-13049045
 ] 

Sylvain Lebresne commented on CASSANDRA-2752:
-

Good catch. +1 (committed).
Thanks Terje.

 repair fails with java.io.EOFException
 --

 Key: CASSANDRA-2752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Terje Marthinussen
Priority: Critical
 Fix For: 0.8.1

 Attachments: 2752.txt


 Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
 INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
 (line 234) Queueing comparison #Differencer #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
 .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
 of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 485) Performing streaming repair of 2 ranges for #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
 fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
 Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
 sections=1 progress=0/36592 - 0%], 1 sstables.
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
 (line 174) Streaming to /1.10.42.82
 ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:9,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 On .82
 ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:12,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-132,5,main]
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.io.EOFException
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at

[jira] [Created] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Sylvain Lebresne (JIRA) 
ConcurrentModificationException in AntiEntropyService.getNeighbors()


 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
 Fix For: 0.8.1




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135402 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/io/sstable/

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 08:16:43 2011
New Revision: 1135402

URL: http://svn.apache.org/viewvc?rev=1135402view=rev
Log:
Merge from 0.8

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)
cassandra/trunk/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 08:16:43 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
 /cassandra/branches/cassandra-0.7:1026516-1133874
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
-/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399
 /cassandra/branches/cassandra-0.8.0:1125021-1130369
 /cassandra/branches/cassandra-0.8.1:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1135402r1=1135401r2=1135402view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Tue Jun 14 08:16:43 2011
@@ -56,6 +56,7 @@
supercolumn tombstone during replica resolution (CASSANDRA-2590)
  * support sstable2json against snapshot sstables (CASSANDRA-2386)
  * remove active-pull schema requests (CASSANDRA-2715)
+ * seek back after deserializing a row to update cache with (CASSANDRA-2752)
 
 
 0.8.0-final

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 08:16:43 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
 /cassandra/branches/cassandra-0.7/contrib:1026516-1133874
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
-/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241
+/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369
 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 08:16:43 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
-/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241
+/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369
 
/cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018
 
/cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 08:16:43 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1133874
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654

[jira] [Updated] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2767:


Attachment: 0001-Fix-ConcurrentModificationException.patch

 ConcurrentModificationException in AntiEntropyService.getNeighbors()
 

 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: repair
 Fix For: 0.8.1

 Attachments: 0001-Fix-ConcurrentModificationException.patch

   Original Estimate: 1h
  Remaining Estimate: 1h



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2767:


Attachment: (was: 0001-Fix-ConcurrentModificationException.patch)

 ConcurrentModificationException in AntiEntropyService.getNeighbors()
 

 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: repair
 Fix For: 0.8.1

 Attachments: 0001-Fix-ConcurrentModificationException.patch

   Original Estimate: 1h
  Remaining Estimate: 1h



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2767:


Attachment: 0001-Fix-ConcurrentModificationException.patch

 ConcurrentModificationException in AntiEntropyService.getNeighbors()
 

 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: repair
 Fix For: 0.8.1

 Attachments: 0001-Fix-ConcurrentModificationException.patch

   Original Estimate: 1h
  Remaining Estimate: 1h



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2759) Scrub could lose increments and replicate that loss

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049065#comment-13049065
 ] 

Jonathan Ellis commented on CASSANDRA-2759:
---

+1

can you add a link to this issue in the dangerous comment?

 Scrub could lose increments and replicate that loss
 ---

 Key: CASSANDRA-2759
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2759
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: counters
 Fix For: 0.8.1

 Attachments: 0001-Don-t-skip-rows-on-scrub-for-counter-CFs.patch, 
 0001-Renew-nodeId-in-scrub-when-skipping-rows.patch


 If scrub cannot 'repair' a corrupted row, it will skip it. On node A, if the 
 row contains some sub-count for A id, those will be lost forever since A is 
 the source of truth on it's current id. We should thus renew node A id when 
 that happens to avoid this (not unlike we do in cleanup).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049066#comment-13049066
 ] 

Jonathan Ellis commented on CASSANDRA-2767:
---

+1

 ConcurrentModificationException in AntiEntropyService.getNeighbors()
 

 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: repair
 Fix For: 0.8.1

 Attachments: 0001-Fix-ConcurrentModificationException.patch

   Original Estimate: 1h
  Remaining Estimate: 1h



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sasha Dolgy (JIRA) 
AntiEntropyService excluding nodes that are on version 0.7 or sooner


 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 

Originally 0.7.6-2 with a Keyspace defined with RF=3

Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
was shut down, new version was turned on, using the existing data files / 
directories and a nodetool repair was run.  
Reporter: Sasha Dolgy


When I run nodetool repair on any of the nodes, the 
/var/log/cassandra/system.log reports errors similar to:

INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
repair because it is on version 0.7 or sooner. You should consider updating 
this node before running repair again.
ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI Runtime]
java.util.ConcurrentModificationException
  at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
  at java.util.HashMap$KeyIterator.next(HashMap.java:828)
  at 
org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
  at 
org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)

The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
suspect that this is because RF=3.  

nodetool ring shows that all nodes are up.  

Client connections (read / write) are not having issues..  

nodetool version on all nodes shows that each node is 0.8.0

At suggestion of some contributors, I have restarted each node and tried to run 
a nodetool repair again ... the result is the same with the messages being 
logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135426 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/db/compaction/CompactionManager.java

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 09:17:00 2011
New Revision: 1135426

URL: http://svn.apache.org/viewvc?rev=1135426view=rev
Log:
Don't skip rows in scub for counter column families
patch by slebresne; reviewed by jbellis for CASSANDRA-2759

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1135426r1=1135425r2=1135426view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jun 14 09:17:00 2011
@@ -52,6 +52,7 @@
  * avoid marking entire list of sstables as actively being compacted
in multithreaded compaction (CASSANDRA-2765)
  * seek back after deserializing a row to update cache with (CASSANDRA-2752)
+ * avoid skipping rows in scrub for counter column family (CASSANDRA-2759)
 
 
 0.8.0-final

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java?rev=1135426r1=1135425r2=1135426view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
 Tue Jun 14 09:17:00 2011
@@ -637,6 +637,7 @@ public class CompactionManager implement
 {
 logger.info(Scrubbing  + sstable);
 CompactionController controller = new CompactionController(cfs, 
Collections.singletonList(sstable), getDefaultGcBefore(cfs), true);
+boolean isCommutative = 
cfs.metadata.getDefaultValidator().isCommutative();
 
 // Calculate the expected compacted filesize
 String compactionFileLocation = 
cfs.table.getDataFileLocation(sstable.length());
@@ -759,6 +760,10 @@ public class CompactionManager implement
 catch (Throwable th2)
 {
 throwIfFatal(th2);
+// Skipping rows is dangerous for counters (see 
CASSANDRA-2759)
+if (isCommutative)
+throw new IOError(th2);
+
 logger.warn(Retry failed too.  Skipping to next 
row (retry's stacktrace follows), th2);
 writer.reset();
 dataFile.seek(nextRowPositionFromIndex);
@@ -767,6 +772,10 @@ public class CompactionManager implement
 }
 else
 {
+// Skipping rows is dangerous for counters (see 
CASSANDRA-2759)
+if (isCommutative)
+throw new IOError(th);
+
 logger.warn(Row at  + dataStart +  is unreadable; 
skipping to next);
 if (currentIndexKey != null)
 dataFile.seek(nextRowPositionFromIndex);

svn commit: r1135427 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/compaction/

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 09:18:31 2011
New Revision: 1135427

URL: http://svn.apache.org/viewvc?rev=1135427view=rev
Log:
Merge from 0.8

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)

cassandra/trunk/src/java/org/apache/cassandra/db/compaction/CompactionManager.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:18:31 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
 /cassandra/branches/cassandra-0.7:1026516-1133874
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
-/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399,1135426
 /cassandra/branches/cassandra-0.8.0:1125021-1130369
 /cassandra/branches/cassandra-0.8.1:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1135427r1=1135426r2=1135427view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Tue Jun 14 09:18:31 2011
@@ -57,6 +57,7 @@
  * support sstable2json against snapshot sstables (CASSANDRA-2386)
  * remove active-pull schema requests (CASSANDRA-2715)
  * seek back after deserializing a row to update cache with (CASSANDRA-2752)
+ * avoid skipping rows in scrub for counter column family (CASSANDRA-2759)
 
 
 0.8.0-final

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:18:31 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
 /cassandra/branches/cassandra-0.7/contrib:1026516-1133874
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
-/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399
+/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399,1135426
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369
 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:18:31 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
-/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399
+/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399,1135426
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369
 
/cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018
 
/cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:18:31 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1133874

[jira] [Assigned] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reassigned CASSANDRA-2768:
-

Assignee: Sylvain Lebresne

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Sylvain Lebresne

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reassigned CASSANDRA-2766:
-

Assignee: Jonathan Ellis

 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis

 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-85,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049078#comment-13049078
 ] 

Sylvain Lebresne commented on CASSANDRA-2768:
-

The important part here is that this is not a repair specific thing per se. The 
important part of the stack trace is the 'Excluding ...' part.
It is triggered because of the following code in AES.getNeighbors:
{noformat}
  if (Gossiper.instance.getVersion(endpoint) = MessagingService.VERSION_07)
  {
  logger.info(Excluding  + endpoint +  from repair because it is on 
version 0.7 or sooner. You should consider updating this node before running 
repair again.);
  neighbors.remove(endpoint);
  }
{noformat}
Since Sasha has reportedly verified that all node report being on 0.8.0, this 
suggests a Gossiper bug that reports the wrong version (even after node 
restarts).

The exception itself has been fixed in CASSANDRA-2767 and should not be the 
focus of attention here.

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Sylvain Lebresne

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2766:
--

Attachment: 2766.txt

looks like a straightforward case of not using threadsafe collections means 
there is no happens-before guarantee where a naive reading of the code would 
expect one (i.e., since buildFutures.add is always called before files.remove, 
after files.isEmpty is true there should be no more changes to 
buildFutures.add).

patch attached that changes both of these from Arraylist to CSLS.

 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
 Attachments: 2766.txt


 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-85,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)

svn commit: r1135441 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/service/AntiEntropyService.java

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 09:46:25 2011
New Revision: 1135441

URL: http://svn.apache.org/viewvc?rev=1135441view=rev
Log:
Fix ConcurrentModificationException in repair
patch by slebresne; reviewed by jbellis for CASSANDRA-2767

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1135441r1=1135440r2=1135441view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jun 14 09:46:25 2011
@@ -53,6 +53,8 @@
in multithreaded compaction (CASSANDRA-2765)
  * seek back after deserializing a row to update cache with (CASSANDRA-2752)
  * avoid skipping rows in scrub for counter column family (CASSANDRA-2759)
+ * fix ConcurrentModificationException in repair when dealing with 0.7 node
+   (CASSANDRA-2767)
 
 
 0.8.0-final

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java?rev=1135441r1=1135440r2=1135441view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java
 Tue Jun 14 09:46:25 2011
@@ -170,12 +170,14 @@ public class AntiEntropyService
 neighbors.remove(FBUtilities.getLocalAddress());
 // Excluding all node with version = 0.7 since they don't know how to
 // create a correct merkle tree (they build it over the full range)
-for (InetAddress endpoint : neighbors)
+IteratorInetAddress iter = neighbors.iterator();
+while (iter.hasNext())
 {
+InetAddress endpoint = iter.next();
 if (Gossiper.instance.getVersion(endpoint) = 
MessagingService.VERSION_07)
 {
 logger.info(Excluding  + endpoint +  from repair because it 
is on version 0.7 or sooner. You should consider updating this node before 
running repair again.);
-neighbors.remove(endpoint);
+iter.remove();
 }
 }
 return neighbors;

svn commit: r1135443 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/service/

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 09:47:30 2011
New Revision: 1135443

URL: http://svn.apache.org/viewvc?rev=1135443view=rev
Log:
Merge from 0.8

Modified:
cassandra/trunk/   (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)

cassandra/trunk/src/java/org/apache/cassandra/service/AntiEntropyService.java

Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:47:30 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
 /cassandra/branches/cassandra-0.7:1026516-1133874
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
-/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399,1135426
+/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1135241,1135399,1135426,1135441
 /cassandra/branches/cassandra-0.8.0:1125021-1130369
 /cassandra/branches/cassandra-0.8.1:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1135443r1=1135442r2=1135443view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Tue Jun 14 09:47:30 2011
@@ -58,6 +58,8 @@
  * remove active-pull schema requests (CASSANDRA-2715)
  * seek back after deserializing a row to update cache with (CASSANDRA-2752)
  * avoid skipping rows in scrub for counter column family (CASSANDRA-2759)
+ * fix ConcurrentModificationException in repair when dealing with 0.7 node
+   (CASSANDRA-2767)
 
 
 0.8.0-final

Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:47:30 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
 /cassandra/branches/cassandra-0.7/contrib:1026516-1133874
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
-/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399,1135426
+/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1135241,1135399,1135426,1135441
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369
 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018
 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:47:30 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
 
/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
-/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399,1135426
+/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1135241,1135399,1135426,1135441
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369
 
/cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018
 
/cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689

Propchange: 
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 09:47:30 2011
@@ -1,7 +1,7 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291

[jira] [Updated] (CASSANDRA-2758) nodetool repair never finishes. Loops forever through merkle trees?

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2758:


Attachment: 0001-Fix-MerkleTree.init-to-not-create-non-sensical-trees.patch

MerkleTree.init(), which is used to create the merkle tree in case there is no 
data, was creating a nonsensical tree by stopping it's iteration too late.

Attached patch to fix (and dumping priority to minor because it has very little 
chance to hit anyone in any real-life situation).

 nodetool repair never finishes. Loops forever through merkle trees?
 ---

 Key: CASSANDRA-2758
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2758
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Sylvain Lebresne
 Fix For: 0.8.1

 Attachments: 
 0001-Fix-MerkleTree.init-to-not-create-non-sensical-trees.patch


 I am not sure all steps here is needed, but as part of testing something 
 else, I set up
 node1: initial_token: 1
 node2: initial_token: 5
 Then:
 {noformat}
 create keyspace myks 
  with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
  with strategy_options = [{ replication_factor:2 }];
 use myks;
 create column family test with comparator = AsciiType and column_metadata=[ 
 {column_name: 'up_', validation_class: LongType, index_type: 0}, 
 {column_name: 'del_', validation_class: LongType, index_type: 0} ]
  and keys_cached = 10 and rows_cached = 1 and 
 min_compaction_threshold = 2;
 quit;
 {noformat}
 Doing nodetool repair after this gets both nodes busy looping forever.
 A quick look at one node in eclipse makes me guess its having fun spinning 
 through  merkle trees, but I have to admit I have not look at it for a long 
 time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2758) nodetool repair never finishes. Loops forever through merkle trees?

2011-06-14 Thread Sylvain Lebresne (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-2758:


Priority: Minor  (was: Major)

 nodetool repair never finishes. Loops forever through merkle trees?
 ---

 Key: CASSANDRA-2758
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2758
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8.1

 Attachments: 
 0001-Fix-MerkleTree.init-to-not-create-non-sensical-trees.patch


 I am not sure all steps here is needed, but as part of testing something 
 else, I set up
 node1: initial_token: 1
 node2: initial_token: 5
 Then:
 {noformat}
 create keyspace myks 
  with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
  with strategy_options = [{ replication_factor:2 }];
 use myks;
 create column family test with comparator = AsciiType and column_metadata=[ 
 {column_name: 'up_', validation_class: LongType, index_type: 0}, 
 {column_name: 'del_', validation_class: LongType, index_type: 0} ]
  and keys_cached = 10 and rows_cached = 1 and 
 min_compaction_threshold = 2;
 quit;
 {noformat}
 Doing nodetool repair after this gets both nodes busy looping forever.
 A quick look at one node in eclipse makes me guess its having fun spinning 
 through  merkle trees, but I have to admit I have not look at it for a long 
 time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2679) Move some column creation logic into Column factory functions

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049118#comment-13049118
 ] 

Sylvain Lebresne commented on CASSANDRA-2679:
-

lgtm but can we rename the 'get' to 'create' (or 'make')  as this better 
suggest what those methods do.

 Move some column creation logic into Column factory functions
 -

 Key: CASSANDRA-2679
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2679
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Stu Hood
Assignee: Stu Hood
Priority: Minor
 Fix For: 1.0

 Attachments: 
 0001-CASSANDRA-2679-Move-Deleted-Expiring-switch-and-contex.txt


 Expiring and Counter columns have extra creation logic that is better 
 encapsulated when implemented inside a factory function.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2221) 'show create' commands on the CLI to export schema

2011-06-14 Thread Pavel Yaskevich (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049120#comment-13049120
 ] 

Pavel Yaskevich commented on CASSANDRA-2221:


needs rebase, tried to apply on cassandra-0.8 branch 
(1e77d805fb893515db6a6ab625297ac17412fb45 latest commit). 

 'show create' commands on the CLI to export schema
 --

 Key: CASSANDRA-2221
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2221
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Reporter: Jeremy Hanna
Assignee: Aaron Morton
Priority: Minor
  Labels: cli
 Fix For: 0.8.1

 Attachments: 0001-add-show-schema-statement-8.patch, 
 0001-add-show-schema-statement.patch


 It would be nice to have 'show create' type of commands on the command-line 
 so that it would generate the DDL for the schema.
 A scenario that would make this useful is where a team works out a data model 
 over time with a dev cluster.  They want to use parts of that schema for new 
 clusters that they create, like a staging/prod cluster.  It would be very 
 handy in this scenario to have some sort of export mechanism.
 Another use case is for testing purposes - you want to replicate a problem.
 We currently have schematool for import/export but that is deprecated and it 
 exports into yaml.
 This new feature would just be able to 'show' - or export if they want the 
 entire keyspace - into a script or commands that could be used in a cli 
 script.  It would need to be able to regenerate everything about the keyspace 
 including indexes and metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.

2011-06-14 Thread Mck SembWever (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13048727#comment-13048727
 ] 

Mck SembWever edited comment on CASSANDRA-2388 at 6/14/11 11:22 AM:


Won't the sorting still be wrong?
For the use-case above it will solve restricting to the correct datacenter, but 
the sorting will still be based on proximity to the wrong node?

bq. I don't think it makes sense to send the client endpoint to this call since 
the endpoint might not be a cassandra node. 
It might not be an alive cassandra node, but it should be a cassandra node. It 
comes from the split's list of endpoints. At least in this use-case, or are you 
referring to general usage for this new api?
bq. It's a reasonable assumption that the endpoint it's talking to is local 
enough to the client to use that.
I don't think so... The endpoint that it talks to is a completely random (just 
the next endpoint listed in the split's list). This is why i think that such 
sorting won't just be wrong but not even close. Does this make sense?

  was (Author: michaelsembwever):
Won't the sorting still be wrong?
For the use-case above it will solve restricting to the correct datacenter, but 
the sorting will still be based on proximity to the wrong node?
  
 ColumnFamilyRecordReader fails for a given split because a host is down, even 
 if records could reasonably be read from other replica.
 -

 Key: CASSANDRA-2388
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2388
 Project: Cassandra
  Issue Type: Bug
  Components: Hadoop
Reporter: Eldon Stegall
Assignee: Mck SembWever
  Labels: hadoop, inputformat
 Fix For: 0.8.1

 Attachments: 0002_On_TException_try_next_split.patch, 
 CASSANDRA-2388.patch, CASSANDRA-2388.patch


 ColumnFamilyRecordReader only tries the first location for a given split. We 
 should try multiple locations for a given split.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.

2011-06-14 Thread T Jake Luciani (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049169#comment-13049169
 ] 

T Jake Luciani edited comment on CASSANDRA-2388 at 6/14/11 1:07 PM:


I think the core issue is you can't assume the hadoop node is running on a 
cassandra node...

If it is then the logic is straight forward, if not then it's possible the 
connection could cross DC boundaries. One possibility is to use the ip octets 
like the RackInferringSnitch.  

How's this proposal then?  keep the sort_endpoints_by_proximity signature as is 
and pass the client endpoint along with the list of data endpoints and add the 
following logic:

1) sort the endpoints using the endpoint_snitch.
2) if client endpoint *is* a valid cassandra node get the nodes DC and prune 
nodes outside of this DC
3) if client endpoint *is not* a valid cassandra node try to infer the DC from 
its ip and prune dataendpoint nodes in a different DC. If no cassandra nodes 
are in the DC list goto 3).
4) all else fails return the sorted endpoint list


  was (Author: tjake):
I think the core issue is you can't assume the hadoop node is running on a 
cassandra node...

If it is then the logic is straight forward, if not then it's possible the 
connection could cross DC boundaries. One possibility is to use the ip octets 
like the RackInferringSnitch.  

How's this proposal then?  keep the sort_endpoints_by_proximity signature as is 
and pass the client endpoint along with the list of data endpoints and add the 
following logic:

1) sort the endpoints using the endpoint_snitch.
2) if client endpoint *is* a valid cassandra node get the nodes DC and prune 
nodes outside of this DC
3) if client endpoint *is not* a valid cassandra node try to infer the DC from 
its ip and prune dataendpoint nodes in a different DC. If no cassandra nodes 
are in the DC list goto 3).
3) all else fails return the sorted endpoint list

  
 ColumnFamilyRecordReader fails for a given split because a host is down, even 
 if records could reasonably be read from other replica.
 -

 Key: CASSANDRA-2388
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2388
 Project: Cassandra
  Issue Type: Bug
  Components: Hadoop
Reporter: Eldon Stegall
Assignee: Mck SembWever
  Labels: hadoop, inputformat
 Fix For: 0.8.1

 Attachments: 0002_On_TException_try_next_split.patch, 
 CASSANDRA-2388.patch, CASSANDRA-2388.patch


 ColumnFamilyRecordReader only tries the first location for a given split. We 
 should try multiple locations for a given split.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.

2011-06-14 Thread T Jake Luciani (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049169#comment-13049169
 ] 

T Jake Luciani commented on CASSANDRA-2388:
---

I think the core issue is you can't assume the hadoop node is running on a 
cassandra node...

If it is then the logic is straight forward, if not then it's possible the 
connection could cross DC boundaries. One possibility is to use the ip octets 
like the RackInferringSnitch.  

How's this proposal then?  keep the sort_endpoints_by_proximity signature as is 
and pass the client endpoint along with the list of data endpoints and add the 
following logic:

1) sort the endpoints using the endpoint_snitch.
2) if client endpoint *is* a valid cassandra node get the nodes DC and prune 
nodes outside of this DC
3) if client endpoint *is not* a valid cassandra node try to infer the DC from 
its ip and prune dataendpoint nodes in a different DC. If no cassandra nodes 
are in the DC list goto 3).
3) all else fails return the sorted endpoint list


 ColumnFamilyRecordReader fails for a given split because a host is down, even 
 if records could reasonably be read from other replica.
 -

 Key: CASSANDRA-2388
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2388
 Project: Cassandra
  Issue Type: Bug
  Components: Hadoop
Reporter: Eldon Stegall
Assignee: Mck SembWever
  Labels: hadoop, inputformat
 Fix For: 0.8.1

 Attachments: 0002_On_TException_try_next_split.patch, 
 CASSANDRA-2388.patch, CASSANDRA-2388.patch


 ColumnFamilyRecordReader only tries the first location for a given split. We 
 should try multiple locations for a given split.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Muga Nishizawa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049170#comment-13049170
 ] 

Muga Nishizawa commented on CASSANDRA-2766:
---

Thanks for your quick response.  But we've encountered the following exception. 
   

ERROR [Thread-18] 2011-06-14 21:54:04,065 AbstractCassandraDaemon.java (line 
113) Fatal exception in thread Thread[Thread-18,5,main]
java.lang.ClassCastException: org.apache.cassandra.streaming.PendingFile cannot 
be cast to java.lang.Comparable
at 
java.util.concurrent.ConcurrentSkipListMap.comparable(ConcurrentSkipListMap.java:621)
at 
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:862)
at 
java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
at 
java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:202)
at 
org.apache.cassandra.streaming.StreamInSession.addFiles(StreamInSession.java:100)
at 
org.apache.cassandra.streaming.IncomingStreamReader.init(IncomingStreamReader.java:49)
at 
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)


 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
 Attachments: 2766.txt


 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at

[jira] [Updated] (CASSANDRA-2758) nodetool repair never finishes. Loops forever through merkle trees?

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2758:
--

Reviewer: stuhood

 nodetool repair never finishes. Loops forever through merkle trees?
 ---

 Key: CASSANDRA-2758
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2758
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8.1

 Attachments: 
 0001-Fix-MerkleTree.init-to-not-create-non-sensical-trees.patch


 I am not sure all steps here is needed, but as part of testing something 
 else, I set up
 node1: initial_token: 1
 node2: initial_token: 5
 Then:
 {noformat}
 create keyspace myks 
  with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
  with strategy_options = [{ replication_factor:2 }];
 use myks;
 create column family test with comparator = AsciiType and column_metadata=[ 
 {column_name: 'up_', validation_class: LongType, index_type: 0}, 
 {column_name: 'del_', validation_class: LongType, index_type: 0} ]
  and keys_cached = 10 and rows_cached = 1 and 
 min_compaction_threshold = 2;
 quit;
 {noformat}
 Doing nodetool repair after this gets both nodes busy looping forever.
 A quick look at one node in eclipse makes me guess its having fun spinning 
 through  merkle trees, but I have to admit I have not look at it for a long 
 time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2766:
--

Attachment: 2766-v2.txt

oops. v2 w/ collections that don't require a comparable.

 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
 Attachments: 2766-v2.txt, 2766.txt


 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-85,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)

--
This message is automatically generated by JIRA.
For more information on JIRA, see:

[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049191#comment-13049191
 ] 

Sylvain Lebresne commented on CASSANDRA-2766:
-

lgtm +1 on v2

 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
 Attachments: 2766-v2.txt, 2766.txt


 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-85,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135611 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/streaming/StreamInSession.java

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 14:33:20 2011
New Revision: 1135611

URL: http://svn.apache.org/viewvc?rev=1135611view=rev
Log:
use threadsafe collections for StreamInSession
patch by jbellis; reviewed by slebresne for CASSANDRA-2766

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1135611r1=1135610r2=1135611view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jun 14 14:33:20 2011
@@ -55,6 +55,7 @@
  * avoid skipping rows in scrub for counter column family (CASSANDRA-2759)
  * fix ConcurrentModificationException in repair when dealing with 0.7 node
(CASSANDRA-2767)
+ * use threadsafe collections for StreamInSession (CASSANDRA-2766)
 
 
 0.8.0-final

Modified: 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java?rev=1135611r1=1135610r2=1135611view=diff
==
--- 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java
 (original)
+++ 
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java
 Tue Jun 14 14:33:20 2011
@@ -24,18 +24,20 @@ import java.util.*;
 import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
 
-import org.apache.cassandra.gms.Gossiper;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.Table;
+import org.apache.cassandra.db.compaction.CompactionManager;
+import org.apache.cassandra.gms.Gossiper;
 import org.apache.cassandra.io.sstable.SSTableReader;
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.utils.Pair;
 import org.cliffc.high_scale_lib.NonBlockingHashMap;
+import org.cliffc.high_scale_lib.NonBlockingHashSet;
 
 /** each context gets its own StreamInSession. So there may be 1 Session per 
host */
 public class StreamInSession
@@ -44,11 +46,11 @@ public class StreamInSession
 
 private static ConcurrentMapPairInetAddress, Long, StreamInSession 
sessions = new NonBlockingHashMapPairInetAddress, Long, StreamInSession();
 
-private final ListPendingFile files = new ArrayListPendingFile();
+private final SetPendingFile files = new 
NonBlockingHashSetPendingFile();
 private final PairInetAddress, Long context;
 private final Runnable callback;
 private String table;
-private final ListFutureSSTableReader buildFutures = new 
ArrayListFutureSSTableReader();
+private final CollectionFutureSSTableReader buildFutures = new 
LinkedBlockingQueueFutureSSTableReader();
 private PendingFile current;
 
 private StreamInSession(PairInetAddress, Long context, Runnable callback)

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sasha Dolgy (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049214#comment-13049214
 ] 

Sasha Dolgy commented on CASSANDRA-2768:


I'd like to add that we are also using Ec2Snitch across the four nodes, 
although, all four nodes are based in the EC2 region of Singapore.  

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Sylvain Lebresne

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reassigned CASSANDRA-2768:
-

Assignee: Brandon Williams  (was: Sylvain Lebresne)

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Brandon Williams

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135638 - in /cassandra/branches/cassandra-0.7: CHANGES.txt src/java/org/apache/cassandra/streaming/StreamInSession.java

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 15:16:04 2011
New Revision: 1135638

URL: http://svn.apache.org/viewvc?rev=1135638view=rev
Log:
backport #2766 from 0.8

Modified:
cassandra/branches/cassandra-0.7/CHANGES.txt

cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/streaming/StreamInSession.java

Modified: cassandra/branches/cassandra-0.7/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/CHANGES.txt?rev=1135638r1=1135637r2=1135638view=diff
==
--- cassandra/branches/cassandra-0.7/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.7/CHANGES.txt Tue Jun 14 15:16:04 2011
@@ -18,6 +18,7 @@
  * fix nodetool ring use with Ec2Snitch (CASSANDRA-2733)
  * fix removing columns and subcolumns that are supressed by a row or
supercolumn tombstone during replica resolution (CASSANDRA-2590)
+ * use threadsafe collections for StreamInSession (CASSANDRA-2766)
 
 
 0.7.6

Modified: 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/streaming/StreamInSession.java
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/streaming/StreamInSession.java?rev=1135638r1=1135637r2=1135638view=diff
==
--- 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/streaming/StreamInSession.java
 (original)
+++ 
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/streaming/StreamInSession.java
 Tue Jun 14 15:16:04 2011
@@ -24,6 +24,7 @@ import java.util.*;
 import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -35,6 +36,7 @@ import org.apache.cassandra.io.sstable.S
 import org.apache.cassandra.net.MessagingService;
 import org.apache.cassandra.utils.Pair;
 import org.cliffc.high_scale_lib.NonBlockingHashMap;
+import org.cliffc.high_scale_lib.NonBlockingHashSet;
 
 /** each context gets its own StreamInSession. So there may be 1 Session per 
host */
 public class StreamInSession
@@ -43,11 +45,11 @@ public class StreamInSession
 
 private static ConcurrentMapPairInetAddress, Long, StreamInSession 
sessions = new NonBlockingHashMapPairInetAddress, Long, StreamInSession();
 
-private final ListPendingFile files = new ArrayListPendingFile();
+private final SetPendingFile files = new 
NonBlockingHashSetPendingFile();
 private final PairInetAddress, Long context;
 private final Runnable callback;
 private String table;
-private final ListFutureSSTableReader buildFutures = new 
ArrayListFutureSSTableReader();
+private final CollectionFutureSSTableReader buildFutures = new 
LinkedBlockingQueueFutureSSTableReader();
 private PendingFile current;
 
 private StreamInSession(PairInetAddress, Long context, Runnable callback)

svn commit: r1135642 - in /cassandra/branches/cassandra-0.8: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 15:21:28 2011
New Revision: 1135642

URL: http://svn.apache.org/viewvc?rev=1135642view=rev
Log:
merge from 0.7

Modified:
cassandra/branches/cassandra-0.8/   (props changed)
cassandra/branches/cassandra-0.8/contrib/   (props changed)

cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
   (props changed)

cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
   (props changed)

cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
   (props changed)

cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
   (props changed)

cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
   (props changed)

Propchange: cassandra/branches/cassandra-0.8/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 15:21:28 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291
-/cassandra/branches/cassandra-0.7:1026516-1133874
+/cassandra/branches/cassandra-0.7:1026516-1133874,1135638
 /cassandra/branches/cassandra-0.7.0:1053690-1055654
 /cassandra/branches/cassandra-0.8:1090934-1125013,1125041
 /cassandra/branches/cassandra-0.8.0:1125021-1130369

Propchange: cassandra/branches/cassandra-0.8/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 15:21:28 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
-/cassandra/branches/cassandra-0.7/contrib:1026516-1133874
+/cassandra/branches/cassandra-0.7/contrib:1026516-1133874,1135638
 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
 /cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125041
 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369

Propchange: 
cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 15:21:28 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
-/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874
+/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1133874,1135638
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654
 
/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125041
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369

Propchange: 
cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 15:21:28 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
-/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1133874
+/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1026516-1133874,1135638
 
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1053690-1055654
 
/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1090934-1125013,1125041
 
/cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:1125021-1130369

Propchange: 
cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Jun 14 15:21:28 2011
@@ -1,5 +1,5 @@
 
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-1052356,1052358-1053452,1053454,1053456-1131291
-/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:1026516-1133874

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sasha Dolgy (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049243#comment-13049243
 ] 

Sasha Dolgy commented on CASSANDRA-2768:


Hi ... able to give more information now:


cassandra:~$ nodetool ring
Address Status State   LoadOwnsToken
   
170141183460469231731687303715884105726
10.128.103.148  Up Normal  961.38 KB   11.22%  
1909554714494251628118265338228798
10.128.94.227   Up Normal  667.56 KB   22.11%  
56713727820156410577229101238628035242
10.128.34.18Up Normal  688.1 KB33.33%  
113427455640312821154458202477256070484
10.128.90.109   Up Normal  965.76 KB   33.33%  
170141183460469231731687303715884105726

Not a lot of data.  I created a new keyspace with (RF=2), dropped the old one.  
Ran repair on the nodes, and now I no longer get the error on some of the 
nodes. 

I can confirm again all systems are reporting:  ReleaseVersion: 0.8.0  from 
'nodetool version'

I am seeing this error on two of the nodes:   

 ERROR [pool-2-thread-14] 2011-06-14 23:33:40,544 CustomTThreadPoolServer.java 
(line 199) Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in 
readMessageBegin, old client?
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
at 
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [pool-2-thread-16] 2011-06-14 23:33:42,024 CustomTThreadPoolServer.java 
(line 199) Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in 
readMessageBegin, old client?
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
at 
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

109 and 148 look to be communicating fine.

18 -- 109 (version error)
18 -- 227 (version error)
227 -- 18 (version error)
227 -- 148 (version error)

For my sanity, I checked and confirmed that all four instances are part of the 
same security group and there are firewall rules allow communication between 
all four nodes on ports 7000 and 9090

Configuration on all nodes is standard with the following exceptions:


#listen_address: localhost
endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch


 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Brandon Williams

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message

svn commit: r1135671 - /cassandra/drivers/java/CHANGES.txt

2011-06-14 Thread slebresne 
Author: slebresne
Date: Tue Jun 14 16:09:31 2011
New Revision: 1135671

URL: http://svn.apache.org/viewvc?rev=1135671view=rev
Log:
Add changelog for java drivers

Added:
cassandra/drivers/java/CHANGES.txt

Added: cassandra/drivers/java/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/drivers/java/CHANGES.txt?rev=1135671view=auto
==
--- cassandra/drivers/java/CHANGES.txt (added)
+++ cassandra/drivers/java/CHANGES.txt Tue Jun 14 16:09:31 2011
@@ -0,0 +1,2 @@
+1.0.4
+  * improve JDBC spec compliance (CASSANDRA-2720)

[jira] [Commented] (CASSANDRA-2741) cli create ks/cf hangs when nodes are in the UNREACHABLE state

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049269#comment-13049269
 ] 

Brandon Williams commented on CASSANDRA-2741:
-

Reported against 0.7.4.

bq. disagree, it's useful for people to know about once the RT bug is fixed.

Fair enough.

 cli create ks/cf hangs when nodes are in the UNREACHABLE state
 --

 Key: CASSANDRA-2741
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2741
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Brandon Williams
Assignee: Jonathan Ellis
 Fix For: 0.7.7


 It still creates the ks/cf, but never returns a prompt afterwards.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2764) generate-eclipse-files still referencing drivers/ source

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049275#comment-13049275
 ] 

Hudson commented on CASSANDRA-2764:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


 generate-eclipse-files still referencing drivers/ source
 

 Key: CASSANDRA-2764
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2764
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Kirk True
Assignee: Kirk True
Priority: Minor
 Fix For: 1.0

 Attachments: trunk-2764.txt


 In trunk, running ant generate-eclipse-files will reference the old drivers 
 top-level directory. The result is that the generated project, once loaded 
 into Eclipse causes errors about the non-existent source directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2752) repair fails with java.io.EOFException

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049271#comment-13049271
 ] 

Hudson commented on CASSANDRA-2752:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


 repair fails with java.io.EOFException
 --

 Key: CASSANDRA-2752
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2752
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
Priority: Critical
 Fix For: 0.8.1

 Attachments: 2752.txt


 Issuing repair on node 1  (1.10.42.81) in a cluster quickly fails with
 INFO [AntiEntropyStage:1] 2011-06-09 19:02:47,999 AntiEntropyService.java 
 (line 234) Queueing comparison #Differencer #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306fb46e, /1
 .10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 468) Endpoints somewhere/1.10.42.81 and /1.10.42.82 have 2 range(s) out 
 of sync for (JP,XXX) on (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,026 AntiEntropyService.java 
 (line 485) Performing streaming repair of 2 ranges for #TreeRequest 
 manual-repair-0c17c5f9-583f-4a31-a6d4-a9e7306
 fb46e, /1.10.42.82, (JP,XXX), (Token(bytes[6e]),Token(bytes[313039])]
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,030 StreamOut.java (line 173) 
 Stream context metadata [/data/cassandra/node0/data/JP/XXX-g-3-Data.db 
 sections=1 progress=0/36592 - 0%], 1 sstables.
  INFO [AntiEntropyStage:1] 2011-06-09 19:02:48,031 StreamOutSession.java 
 (line 174) Streaming to /1.10.42.82
 ERROR [CompactionExecutor:9] 2011-06-09 19:02:48,970 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:9,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 On .82
 ERROR [CompactionExecutor:12] 2011-06-09 19:02:48,051 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[CompactionExecutor:12,1,main]
 java.io.EOFException
 at java.io.RandomAccessFile.readInt(RandomAccessFile.java:725)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.doIndexing(SSTableWriter.java:457)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:364)
 at 
 org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1099)
 at 
 org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:1090)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 ERROR [Thread-132] 2011-06-09 19:02:48,051 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-132,5,main]
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.io.EOFException
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:152)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 Caused by: java.util.concurrent.ExecutionException: java.io.EOFException
 at

[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException during node recovery

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049273#comment-13049273
 ] 

Hudson commented on CASSANDRA-2766:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])
use threadsafe collections for StreamInSession
patch by jbellis; reviewed by slebresne for CASSANDRA-2766

jbellis : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1135611
Files : 
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* 
/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java


 ConcurrentModificationException during node recovery
 

 Key: CASSANDRA-2766
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Terje Marthinussen
Assignee: Jonathan Ellis
 Attachments: 2766-v2.txt, 2766.txt


 Testing some node recovery operations.
 In this case:
 1. Data is being added/updated as it would in production
 2. repair is running on other nodes in the cluster
 3. we wiped data on this node and started up again, but before repair was 
 actually started on this node (but it had gotten data through the regular 
 data feed) we got this error.
 I see no indication in the logs that outgoing streams has been started, but 
 the node have finished one incoming stream before this (I guess from some 
 other node doing repair).
  INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
  INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java 
 (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 302) Started hinted handoff for endpoint /1.10.42.216
  INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java 
 (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
  INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 
 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
 ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-84,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-79,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-83,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at 
 org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
 at 
 org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
 ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 
 113) Fatal exception in thread Thread[Thread-85,5,main]
 java.util.ConcurrentModificationException
 at 
 java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
 at java.util.AbstractList$Itr.next(AbstractList.java:343)
 at 
 org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
 at

[jira] [Commented] (CASSANDRA-2767) ConcurrentModificationException in AntiEntropyService.getNeighbors()

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049272#comment-13049272
 ] 

Hudson commented on CASSANDRA-2767:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


 ConcurrentModificationException in AntiEntropyService.getNeighbors()
 

 Key: CASSANDRA-2767
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2767
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: repair
 Fix For: 0.8.1

 Attachments: 0001-Fix-ConcurrentModificationException.patch

   Original Estimate: 1h
  Remaining Estimate: 1h



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049274#comment-13049274
 ] 

Hudson commented on CASSANDRA-2765:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


 DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
 --

 Key: CASSANDRA-2765
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2765
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Benjamin Coverston
Assignee: Benjamin Coverston
 Fix For: 0.8.1

 Attachments: 
 0001-removed-code-adding-all-sstables-to-the-view-builder.patch, 2765-v2.txt

   Original Estimate: 1h
  Remaining Estimate: 1h

 At some point if the list isn't cleaned up with this symptom compactions will 
 stop until the server is restarted.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2759) Scrub could lose increments and replicate that loss

2011-06-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049276#comment-13049276
 ] 

Hudson commented on CASSANDRA-2759:
---

Integrated in Cassandra-0.8 #170 (See 
[https://builds.apache.org/job/Cassandra-0.8/170/])


 Scrub could lose increments and replicate that loss
 ---

 Key: CASSANDRA-2759
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2759
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
  Labels: counters
 Fix For: 0.8.1

 Attachments: 0001-Don-t-skip-rows-on-scrub-for-counter-CFs.patch, 
 0001-Renew-nodeId-in-scrub-when-skipping-rows.patch


 If scrub cannot 'repair' a corrupted row, it will skip it. On node A, if the 
 row contains some sub-count for A id, those will be lost forever since A is 
 the source of truth on it's current id. We should thus renew node A id when 
 that happens to avoid this (not unlike we do in cleanup).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2614) create Column and CounterColumn in the same column family

2011-06-14 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049308#comment-13049308
 ] 

Sylvain Lebresne commented on CASSANDRA-2614:
-

Turns out there is a tiny hiccup.
For a Deletion (inside a thrift Mutation), the timestamp is required to be set 
if the CF is a regular one, but not if it is a counter CF. But more 
importantly, for counter CF, the timestamp should be a server generated 
timestamp.

If we allow row to mix counters and regular columns, then when facing a 
Deletion for a full row, there is no way to accommodate those two requirements. 
It would kind of be a shame of not doing this because of Deletion, but I don't 
see a good way around this (other than changing the API, which would move that 
ticket to 1.0).

Ideas ? 

 create Column and CounterColumn in the same column family
 -

 Key: CASSANDRA-2614
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Dave Rav
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8.1


 create Column and CounterColumn in the same column family

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2427:


Attachment: CASSANDRA-2427-4-trunk.txt

Make CommitLog implement MBean
Executor no longer implement MBean
Wire the logic for removing the oldest segment


 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Patricio Echague (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049318#comment-13049318
 ] 

Patricio Echague edited comment on CASSANDRA-2427 at 6/14/11 6:10 PM:
--

Make CommitLog implement MBean
Executor no longer implement MBean
Wire the logic for removing the oldest segment
CommitLogTotalSize is exposed in bytes now


  was (Author: patricioe):
Make CommitLog implement MBean
Executor no longer implement MBean
Wire the logic for removing the oldest segment

  
 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2427:


Attachment: (was: CASSANDRA-2427-3-trunk-addNewSetting.txt)

 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2427:


Attachment: CASSANDRA-2427-5-trunk.txt

Add entry to CHANGES.txt

 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, CASSANDRA-2427-5-trunk.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Alan Liang (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049323#comment-13049323
 ] 

Alan Liang commented on CASSANDRA-2753:
---

Makes sense, I'll move the tracking outside of the serializer. However, one 
thing I realized that I missed is to also capture max timestamp of counter data 
being streamed over from the other nodes. The challenge is where to capture the 
max timestamp without doing it within the AbstractedCompactedRow#write method. 
But it seems like I have no choice without sacrificing performance by iterating 
over the file again to collect the max timestamp. This is because a 
LazilyCompactedRow keeps only a single column in memory and this only happens 
within the write method. What do you think?

 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049326#comment-13049326
 ] 

Brandon Williams commented on CASSANDRA-2768:
-

bq. Since Sasha has reportedly verified that all node report being on 0.8.0, 
this suggests a Gossiper bug that reports the wrong version (even after node 
restarts).

Gossiper's setVersion and getVersion are fairly straightforward, and setVersion 
is called in IncomingTcpConnection and setting it to whatever the remote node 
said to, so a bug here looks unlikely.  The version information is not 
persisted anywhere, so the remote node has to be indicating it is 0.7.  I was 
unable to reproduce following Sasha's steps, so I think the most likely 
explanation here is that a node is mistakenly still on 0.7.

{quote}
I am seeing this error on two of the nodes:

ERROR [pool-2-thread-14] 2011-06-14 23:33:40,544 CustomTThreadPoolServer.java 
(line 199) Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in 
readMessageBegin, old client?
{quote}

This is indicative of a client-side thrift compatibility problem and is 
unrelated, as thrift is not used for internode communication.

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Brandon Williams

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2614) create Column and CounterColumn in the same column family

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049357#comment-13049357
 ] 

Jonathan Ellis commented on CASSANDRA-2614:
---

I'd be fine with saying it's only supported from CQL.  It's not worth breaking 
Thrift compatibility over.

 create Column and CounterColumn in the same column family
 -

 Key: CASSANDRA-2614
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2614
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Dave Rav
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8.1


 create Column and CounterColumn in the same column family

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049366#comment-13049366
 ] 

Brandon Williams commented on CASSANDRA-2491:
-

Why the changes to StreamRequestMessage?

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2641) AbstractBounds.normalize should deal with overlapping ranges

2011-06-14 Thread Stu Hood (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049369#comment-13049369
 ] 

Stu Hood commented on CASSANDRA-2641:
-

bq. getRestrictedRanges splits a range at different tokens.
You're right... that was my imagination.

+1 looks good!

 AbstractBounds.normalize should deal with overlapping ranges
 

 Key: CASSANDRA-2641
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2641
 Project: Cassandra
  Issue Type: Test
  Components: Core
Reporter: Stu Hood
Assignee: Stu Hood
Priority: Minor
 Fix For: 0.8.1

 Attachments: 0001-Assert-non-overlapping-ranges-in-normalize.txt, 
 0001-Make-normalize-deoverlap-ranges.patch, 
 0002-Don-t-use-overlapping-ranges-in-tests.txt


 Apparently no consumers have encountered it in production, but 
 AbstractBounds.normalize does not handle overlapping ranges. If given 
 overlapping ranges, the output will be sorted but still overlapping, for 
 which SSTableReader.getPositionsForRanges will choose ranges in an SSTable 
 that may overlap.
 We should either add an assert in normalize(), or in getPositionsForRanges() 
 to ensure that this never bites us in production.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Vijay (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049370#comment-13049370
 ] 

Vijay commented on CASSANDRA-2491:
--

Ooops was not intentional (except changing it to getBCA() because of the 
rebase svn patch created a mess... i will rebase it again...

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2679) Move some column creation logic into Column factory functions

2011-06-14 Thread Stu Hood (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stu Hood updated CASSANDRA-2679:


Attachment: 0001-CASSANDRA-2679-Move-Deleted-Expiring-switch-and-contex.txt

Fixed, thanks!

 Move some column creation logic into Column factory functions
 -

 Key: CASSANDRA-2679
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2679
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Stu Hood
Assignee: Stu Hood
Priority: Minor
 Fix For: 1.0

 Attachments: 
 0001-CASSANDRA-2679-Move-Deleted-Expiring-switch-and-contex.txt


 Expiring and Counter columns have extra creation logic that is better 
 encapsulated when implemented inside a factory function.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2758) nodetool repair never finishes. Loops forever through merkle trees?

2011-06-14 Thread Stu Hood (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049383#comment-13049383
 ] 

Stu Hood commented on CASSANDRA-2758:
-

+1, thanks!

 nodetool repair never finishes. Loops forever through merkle trees?
 ---

 Key: CASSANDRA-2758
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2758
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Terje Marthinussen
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 0.8.1

 Attachments: 
 0001-Fix-MerkleTree.init-to-not-create-non-sensical-trees.patch


 I am not sure all steps here is needed, but as part of testing something 
 else, I set up
 node1: initial_token: 1
 node2: initial_token: 5
 Then:
 {noformat}
 create keyspace myks 
  with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
  with strategy_options = [{ replication_factor:2 }];
 use myks;
 create column family test with comparator = AsciiType and column_metadata=[ 
 {column_name: 'up_', validation_class: LongType, index_type: 0}, 
 {column_name: 'del_', validation_class: LongType, index_type: 0} ]
  and keys_cached = 10 and rows_cached = 1 and 
 min_compaction_threshold = 2;
 quit;
 {noformat}
 Doing nodetool repair after this gets both nodes busy looping forever.
 A quick look at one node in eclipse makes me guess its having fun spinning 
 through  merkle trees, but I have to admit I have not look at it for a long 
 time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker

2011-06-14 Thread Benjamin Coverston (JIRA) 
Cannot Create Duplicate Compaction Marker
-

 Key: CASSANDRA-2769
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Benjamin Coverston


Concurrent compaction can trigger the following exception: I'll attach a patch 
to fix this.

java.io.IOError: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
at 
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294)
at 
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255)
at 
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932)
at 
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634)
... 12 more

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049388#comment-13049388
 ] 

Jonathan Ellis commented on CASSANDRA-2753:
---

bq. also capture max timestamp of counter data being streamed over from the 
other nodes

is this really counter-specific or does it affect all streamed data?

 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2770) Expose cassandra.yaml location though jmx

2011-06-14 Thread Mike Bulman (JIRA) 
Expose cassandra.yaml location though jmx
-

 Key: CASSANDRA-2770
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2770
 Project: Cassandra
  Issue Type: New Feature
Reporter: Mike Bulman
 Fix For: 0.7.7, 0.8.1


Going to implement this right away to get into next versons.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-2769:
--

Description: 
Concurrent compaction can trigger the following exception when two threads 
compact the same sstable. DataTracker attempts to prevent this but apparently 
not successfully.

java.io.IOError: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
at 
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294)
at 
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255)
at 
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932)
at 
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634)
... 12 more

  was:
Concurrent compaction can trigger the following exception: I'll attach a patch 
to fix this.

java.io.IOError: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
at 
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294)
at 
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255)
at 
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932)
at 
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634)
... 12 more

   Assignee: Sylvain Lebresne

 Cannot Create Duplicate Compaction Marker
 -

 Key: CASSANDRA-2769
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Benjamin Coverston
Assignee: Sylvain Lebresne

 Concurrent compaction can trigger the following exception when two threads 
 compact the same sstable. DataTracker attempts to prevent this but apparently 
 not successfully.
 java.io.IOError: java.io.IOException: Unable to create compaction marker
   at 
 org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
   at 
 org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321)
   at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294)
   at 
 org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255)
   at 
 org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932)
   at 
 org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119)
   at 
 org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sasha Dolgy (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049411#comment-13049411
 ] 

Sasha Dolgy commented on CASSANDRA-2768:


I can assure you they are all on 0.8.0 if the output of nodetool version is 
correct

cassandra@ip-10-128-34-18:~$ nodetool -h 10.128.34.18 -p 9090 ring
Address Status State   LoadOwnsToken
   
170141183460469231731687303715884105726
10.128.103.148  Up Normal  1.02 MB 11.22%  
1909554714494251628118265338228798
10.128.94.227   Up Normal  667.56 KB   22.11%  
56713727820156410577229101238628035242
10.128.34.18Up Normal  688.1 KB33.33%  
113427455640312821154458202477256070484
10.128.90.109   Up Normal  1.11 MB 33.33%  
170141183460469231731687303715884105726
cassandra@ip-10-128-34-18:~$

cassandra@ip-10-128-34-18:~$ nodetool -h 10.128.34.18 -p 9090 version
ReleaseVersion: 0.8.0
cassandra@ip-10-128-34-18:~$


cassandra@ip-10-128-94-227:~$ nodetool -h 10.128.94.227 -p 9090 version
ReleaseVersion: 0.8.0
cassandra@ip-10-128-94-227:~$


cassandra@ip-10-128-90-109:~$ nodetool -h 10.128.90.109 -p 9090 version
ReleaseVersion: 0.8.0
cassandra@ip-10-128-90-109:~$


cassandra@ip-10-128-103-148:~$ nodetool -h 10.128.103.148 -p 9090 version
ReleaseVersion: 0.8.0
cassandra@ip-10-128-103-148:~$


 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Brandon Williams

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Alan Liang (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049413#comment-13049413
 ] 

Alan Liang commented on CASSANDRA-2753:
---

I already have a solution to capture max timestamp for non counter data as seen 
in the current patch. So this really is only a problem for streamed counter 
data. 

 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2770) Expose data_dir though jmx

2011-06-14 Thread Mike Bulman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Bulman updated CASSANDRA-2770:
---

Summary: Expose data_dir though jmx  (was: Expose cassandra.yaml location 
though jmx)

 Expose data_dir though jmx
 --

 Key: CASSANDRA-2770
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2770
 Project: Cassandra
  Issue Type: New Feature
Reporter: Mike Bulman
 Fix For: 0.7.7, 0.8.1


 Going to implement this right away to get into next versons.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049417#comment-13049417
 ] 

Jonathan Ellis commented on CASSANDRA-2753:
---

Can you elaborate?

 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of NodeTool by DavidAllsopp

2011-06-14 Thread Apache Wiki 
Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The NodeTool page has been changed by DavidAllsopp:
http://wiki.apache.org/cassandra/NodeTool?action=diffrev1=13rev2=14

Comment:
Updated nodetool usage output to ouput from v0.7.4 - may need updating further 
if this has changed for later versions...

  
  ''Note: This utility currently requires the same environment as cassandra 
itself, namely the same classpath (including log4j.properties), and a valid 
storage-conf property.''
  
- Running `bin/nodetool` with no arguments produces some simple usage output.
+ Running `bin/nodetool` with no arguments produces some usage output.
  
  {{{
+ Starting NodeTool
+ Missing required option: h
- usage: java org.apache.cassandra.tools.NodeCmd -host arg command
+ usage: java org.apache.cassandra.tools.NodeCmd --host arg command
  
-  -h, -host arg   node hostname or ip address
+  -h,--host argnode hostname or ip address
-  -p, -port arg   remote jmx agent port number
+  -p,--port argremote jmx agent port number
+  -pw,--password arg   remote jmx agent password
+  -u,--username argremote jmx agent username
  
- Available commands: ring, info, version, cleanup, compact, cfstats,
- snapshot [snapshotname], clearsnapshot, tpstats, flush, drain, repair,
- decommission, move, loadbalance, removetoken, setcachecapacity keyspace
- cfname keycachecapacity rowcachecapacity, getcompactionthreshold,
- setcompactionthreshold [minthreshold] ([maxthreshold]), streams [host]
+ Available commands:
+   ring   - Print informations on the token ring
+   join   - Join the ring
+   info   - Print node informations (uptime, load, ...)
+   cfstats- Print statistics on column families
+   clearsnapshot  - Remove all existing snapshots
+   version- Print cassandra version
+   tpstats- Print usage statistics of thread pools
+   drain  - Drain the node (stop accepting writes and flush 
all column families)
+   decommission   - Decommission the node
+   loadbalance- Loadbalance the node
+   compactionstats- Print statistics on compactions
+   disablegossip  - Disable gossip (effectively marking the node dead)
+   enablegossip   - Reenable gossip
+   disablethrift  - Disable thrift server
+   enablethrift   - Reenable thrift server
+   snapshot [snapshotname] - Take a snapshot using optional name snapshotname
+   netstats [host]- Print network information on provided host 
(connecting node by default)
+   move new token   - Move node on the token ring to a new token
+   removetoken status|force|token - Show status of current token removal, 
force completion of pending removal or remove providen token
+   flush [keyspace] [cfnames] - Flush one or more column family
+   repair [keyspace] [cfnames] - Repair one or more column family
+   cleanup [keyspace] [cfnames] - Run cleanup on one or more column family
+   compact [keyspace] [cfnames] - Force a (major) compaction on one or more 
column family
+   scrub [keyspace] [cfnames] - Scrub (rebuild sstables for) one or more 
column family
+   invalidatekeycache [keyspace] [cfnames] - Invalidate the key cache of one 
or more column family
+   invalidaterowcache [keyspace] [cfnames] - Invalidate the key cache of one 
or more column family
+   getcompactionthreshold keyspace cfname - Print min and max compaction 
thresholds for a given column family
+   cfhistograms keyspace cfname - Print statistic histograms for a given 
column family
+   setcachecapacity keyspace cfname keycachecapacity rowcachecapacity 
- Set the key and row cache capacities of a given column family
+   setcompactionthreshold keyspace cfname minthreshold maxthreshold - 
Set the min and max compaction thresholds for a given column family
  }}}
  The -host argument is required, the -port argument is optional and will 
default to 8080 if not supplied.

[jira] [Commented] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Sasha Dolgy (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049421#comment-13049421
 ] 

Sasha Dolgy commented on CASSANDRA-2768:


After dropping the old keyspace, and creating the new keyspace .. I didn't 
cycle through and stop / start each node.  Have just done that now and all 
appears to be fine.  No more errors in the logs.  This problem was 100% there, 
on version 0.8.0 with the old keyspace.  

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy
Assignee: Brandon Williams

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Cassandra Wiki] Update of NodeTool by DavidAllsopp

2011-06-14 Thread Apache Wiki 
Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The NodeTool page has been changed by DavidAllsopp:
http://wiki.apache.org/cassandra/NodeTool?action=diffrev1=14rev2=15

Comment:
Added sections on nodetool flush and scrub

  }}}
  The -host argument is required, the -port argument is optional and will 
default to 8080 if not supplied.
  
- * In 0.7, streams is replaced by netstats. 
  == Ring ==
  The ring command will present node status and an ascii art rendition of the 
ring, ''as determined by the node being queried.''
  
@@ -98, +97 @@

  == Drain ==
  Flushes memtables on the node and stop accepting writes. Reads will still be 
processed. Useful for rolling upgrades.
  
+ == Flush ==
+ Flushes memtables (in memory) to SSTables (on disk), which also enables 
CommitLog segments to be deleted. 
+ 
+ == Scrub ==
+ Cassandra v0.7.1 and v0.7.2 shipped with a bug that caused incorrect 
row-level bloom filters to be generated when compacting sstables generated with 
earlier versions.  This would manifest in IOExceptions during column name-based 
queries.  v0.7.3 provides nodetool scrub to rebuild sstables with correct 
bloom filters, with no data lost. (If your cluster was never on 0.7.0 or 
earlier, you don't have to worry about this.)  Note that nodetool scrub will 
snapshot your data files before rebuilding, just in case.
+

[jira] [Updated] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2427:


Attachment: CASSANDRA-2427-6-trunk-explainPropBetter.txt

add a better explanation in yaml file of how much space the commit log can grow.

 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, CASSANDRA-2427-5-trunk.txt, 
 CASSANDRA-2427-6-trunk-explainPropBetter.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Patricio Echague (JIRA) 
 Remove the commitlog segment size
--

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
 Fix For: 1.0


Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135809 - in /cassandra/trunk: ./ src/java/org/apache/cassandra/config/ src/java/org/apache/cassandra/db/commitlog/

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 21:39:47 2011
New Revision: 1135809

URL: http://svn.apache.org/viewvc?rev=1135809view=rev
Log:
add commitlog_total_space_in_mb
patch by Patricio Echague; reviewed by jbellis for CASSANDRA-2427

Added:

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java
Removed:

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogExecutorServiceMBean.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/PeriodicCommitLogExecutorServiceMBean.java
Modified:
cassandra/trunk/CHANGES.txt
cassandra/trunk/src/java/org/apache/cassandra/config/Config.java
cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogExecutorService.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogExecutorService.java
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLog.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/ICommitLogExecutorService.java

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/PeriodicCommitLogExecutorService.java

Modified: cassandra/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1135809r1=1135808r2=1135809view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Tue Jun 14 21:39:47 2011
@@ -1,5 +1,6 @@
 1.0-dev
  * removed binarymemtable (CASSANDRA-2692)
+ * add commitlog_total_space_in_mb to prevent fragmented logs (CASSANDRA-2427)
 
 
 0.8.1

Modified: cassandra/trunk/src/java/org/apache/cassandra/config/Config.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/config/Config.java?rev=1135809r1=1135808r2=1135809view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/config/Config.java (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/config/Config.java Tue Jun 14 
21:39:47 2011
@@ -58,6 +58,7 @@ public class Config
 
 public Integer memtable_flush_writers = null; // will get set to the 
length of data dirs in DatabaseDescriptor
 public Integer memtable_total_space_in_mb;
+public Integer commitlog_total_space_in_mb = 4096;
 
 public Integer sliced_buffer_size_in_kb = 64;
 

Modified: 
cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java?rev=1135809r1=1135808r2=1135809view=diff
==
--- 
cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
(original)
+++ 
cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java 
Tue Jun 14 21:39:47 2011
@@ -1075,4 +1075,9 @@ public class DatabaseDescriptor
 {
 return conf.memtable_total_space_in_mb  0;
 }
+
+public static long getTotalCommitlogSpaceInMB()
+{
+return conf.commitlog_total_space_in_mb;
+}
 }

Modified: 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogExecutorService.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogExecutorService.java?rev=1135809r1=1135808r2=1135809view=diff
==
--- 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogExecutorService.java
 (original)
+++ 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogExecutorService.java
 Tue Jun 14 21:39:47 2011
@@ -32,27 +32,6 @@ public abstract class AbstractCommitLogE
 {
 protected volatile long completedTaskCount = 0;
 
-protected static void registerMBean(Object o)
-{
-MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
-try
-{
-mbs.registerMBean(o, new 
ObjectName(org.apache.cassandra.db:type=Commitlog));
-}
-catch (Exception e)
-{
-throw new RuntimeException(e);
-}
-}
-
-/**
- * Get the current number of running tasks
- */
-public int getActiveCount()
-{
-return 1;
-}
-
 /**
  * Get the number of completed tasks
  */

Modified: 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogExecutorService.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogExecutorService.java?rev=1135809r1=1135808r2=1135809view=diff
==
---

buildbot failure in ASF Buildbot on cassandra-trunk

2011-06-14 Thread buildbot 
The Buildbot has detected a new failure on builder cassandra-trunk while 
building ASF Buildbot.
Full details are available at:
 http://ci.apache.org/builders/cassandra-trunk/builds/1364

Buildbot URL: http://ci.apache.org/

Buildslave for this Build: isis_ubuntu

Build Reason: scheduler
Build Source Stamp: [branch cassandra/trunk] 1135809
Blamelist: jbellis

BUILD FAILED: failed compile

sincerely,
 -The Buildbot

svn commit: r1135810 - /cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 21:40:44 2011
New Revision: 1135810

URL: http://svn.apache.org/viewvc?rev=1135810view=rev
Log:
fix doubly-created file

Modified:

cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java

Modified: 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java?rev=1135810r1=1135809r2=1135810view=diff
==
--- 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java 
(original)
+++ 
cassandra/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogMBean.java 
Tue Jun 14 21:40:44 2011
@@ -36,43 +36,4 @@ public interface CommitLogMBean
  * Get the current size used by all the commitlog segments.
  */
 public long getTotalCommitlogSize();
-}
-/*
-* Licensed to the Apache Software Foundation (ASF) under one
-* or more contributor license agreements.  See the NOTICE file
-* distributed with this work for additional information
-* regarding copyright ownership.  The ASF licenses this file
-* to you under the Apache License, Version 2.0 (the
-* License); you may not use this file except in compliance
-* with the License.  You may obtain a copy of the License at
-*
-*http://www.apache.org/licenses/LICENSE-2.0
-*
-* Unless required by applicable law or agreed to in writing,
-* software distributed under the License is distributed on an
-* AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-* KIND, either express or implied.  See the License for the
-* specific language governing permissions and limitations
-* under the License.
-*/
-package org.apache.cassandra.db.commitlog;
-
-
-
-public interface CommitLogMBean
-{
-/**
- * Get the number of completed tasks
- */
-public long getCompletedTasks();
-
-/**
- * Get the number of tasks waiting to be executed
- */
-public long getPendingTasks();
-
-/**
- * Get the current size used by all the commitlog segments.
- */
-public long getTotalCommitlogSize();
-}
+}
\ No newline at end of file

[jira] [Issue Comment Edited] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Alan Liang (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049444#comment-13049444
 ] 

Alan Liang edited comment on CASSANDRA-2753 at 6/14/11 9:40 PM:


There are basically 3 places where we need to track max timestamps:

1. Memtable flush
2. During compaction (we simply take the max timestamp already recorded for the 
sstables)
3. Streamed data (normal columns and counter columns)

The challenge here is to capture the max timestamp for newly streamed data. 

For non-counter streamed data, RowIndexer#doIndexing goes through the streamed 
data files and simply updates the cache for the new rows. It iterates over the 
column families without deserializing the columns. To capture max timestamp 
here, I actually deserialize the columns from disk. This incurs more CPU but 
since it is already doing disk seeks when calling  
deserializeFromSSTableNoColumns(), the seek is less costly.

For counter streamed data, CommutativeRowIndexer#doIndexing actually creates 
new data files from the streamed data files. It does this by building an 
AbstractCompactedRow which can be either PreCompactedRow or LazilyCompactedRow. 
Collecting the max timestamp for PreCompactedRow is easy since all the columns 
are in memory. For LazilyCompactedRow, the only place where I can observe the 
max timestamp is during the #write method. Capturing the max timestamp inside 
#write is obviously not ideal since it would introduce a side effect. 
Alternatively, I could capture the max timestamp by deserializing the entire 
LazilyCompactedRow again but this obviously would mean more IO/CPU.

So it looks like I have to capture the max timestamp inside #write.

  was (Author: alanliang):
There are basically 3 places where we need to track max timestamps:

1. Memtable flush
2. During compaction (we simply take the max timestamp already recorded for the 
sstables)
3. Streamed data (normal columns and counter columns)

The challenge here is to capture the max timestamp for newly streamed data. 

For non-counter streamed data, RowIndexer#doIndexing goes through the streamed 
data files and simply updates the cache for the new rows. It iterates over the 
column families without deserializing the columns. To capture max timestamp 
here, I actually deserialize the columns from disk. This incurs more CPU but 
since it is already doing disk seeks when calling  
deserializeFromSSTableNoColumns(), the seek is less costly.

For counter streamed data, CommutativeRowIndexer#doIndexing actually creates 
new data files from the streamed data files. It does this by building an 
AbstractCompactedRow which can be either PreCompactedRow or LazilyCompactedRow. 
Collecting the max timestamp for PreCompactedRow is easy since all the columns 
are in memory. For LazilyCompactedRow, the only place where I can observe the 
max timestamp is during the #write method. Capturing the max timestamp is 
obviously not ideal since it would introduce a side effect. Alternatively, I 
could capture the max timestamp by deserializing the entire LazilyCompactedRow 
again but this obviously would mean more IO/CPU.

So it looks like I have to capture the max timestamp inside #write.
  
 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2753) Capture the max client timestamp for an SSTable

2011-06-14 Thread Alan Liang (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049444#comment-13049444
 ] 

Alan Liang commented on CASSANDRA-2753:
---

There are basically 3 places where we need to track max timestamps:

1. Memtable flush
2. During compaction (we simply take the max timestamp already recorded for the 
sstables)
3. Streamed data (normal columns and counter columns)

The challenge here is to capture the max timestamp for newly streamed data. 

For non-counter streamed data, RowIndexer#doIndexing goes through the streamed 
data files and simply updates the cache for the new rows. It iterates over the 
column families without deserializing the columns. To capture max timestamp 
here, I actually deserialize the columns from disk. This incurs more CPU but 
since it is already doing disk seeks when calling  
deserializeFromSSTableNoColumns(), the seek is less costly.

For counter streamed data, CommutativeRowIndexer#doIndexing actually creates 
new data files from the streamed data files. It does this by building an 
AbstractCompactedRow which can be either PreCompactedRow or LazilyCompactedRow. 
Collecting the max timestamp for PreCompactedRow is easy since all the columns 
are in memory. For LazilyCompactedRow, the only place where I can observe the 
max timestamp is during the #write method. Capturing the max timestamp is 
obviously not ideal since it would introduce a side effect. Alternatively, I 
could capture the max timestamp by deserializing the entire LazilyCompactedRow 
again but this obviously would mean more IO/CPU.

So it looks like I have to capture the max timestamp inside #write.

 Capture the max client timestamp for an SSTable
 ---

 Key: CASSANDRA-2753
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2753
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Alan Liang
Assignee: Alan Liang
Priority: Minor
 Attachments: 
 0003-capture-max-timestamp-for-sstable-and-introduced-SST.patch




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2427) Heuristic or hard cap to prevent fragmented commit logs from bringing down the server

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2427.
---

Resolution: Fixed
  Reviewer: jbellis

committed, thanks!

 Heuristic or hard cap to prevent fragmented commit logs from bringing down 
 the server
 -

 Key: CASSANDRA-2427
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2427
 Project: Cassandra
  Issue Type: Improvement
Reporter: Benjamin Coverston
Assignee: Patricio Echague
  Labels: commitlog, hardening
 Fix For: 1.0

 Attachments: CASSANDRA-2427-4-trunk.txt, CASSANDRA-2427-5-trunk.txt, 
 CASSANDRA-2427-6-trunk-explainPropBetter.txt, 
 CASSANDRA-2427-Jconsole-snapshot.tiff, CASSANDRA-2427-trunk.txt


 Widely divergent write rates on column families can cause the commit log 
 segments to fragment. In some cases we have seen the commit log partition 
 overrun.
 One solution here would be to create a heuristic for segment fragmentation to 
 trigger a flush (commit log segments/memtable) or simply track the free disk 
 space and force a global flush when the disk gets to 80% capacity.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (CASSANDRA-2768) AntiEntropyService excluding nodes that are on version 0.7 or sooner

2011-06-14 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis resolved CASSANDRA-2768.
---

Resolution: Cannot Reproduce
  Assignee: (was: Brandon Williams)

 AntiEntropyService excluding nodes that are on version 0.7 or sooner
 

 Key: CASSANDRA-2768
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2768
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
 Environment: 4 node environment -- 
 Originally 0.7.6-2 with a Keyspace defined with RF=3
 Upgraded all nodes ( 1 at a time ) to version 0.8.0:  For each node, the node 
 was shut down, new version was turned on, using the existing data files / 
 directories and a nodetool repair was run.  
Reporter: Sasha Dolgy

 When I run nodetool repair on any of the nodes, the 
 /var/log/cassandra/system.log reports errors similar to:
 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AntiEntropyService.java (line 177) Excluding /10.128.34.18 from 
 repair because it is on version 0.7 or sooner. You should consider updating 
 this node before running repair again.
 ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13 
 21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception in 
 thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI 
 Runtime]
 java.util.ConcurrentModificationException
   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
   at java.util.HashMap$KeyIterator.next(HashMap.java:828)
   at 
 org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
   at 
 org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)
 The INFO message and subsequent ERROR message are logged for 2 nodes .. I 
 suspect that this is because RF=3.  
 nodetool ring shows that all nodes are up.  
 Client connections (read / write) are not having issues..  
 nodetool version on all nodes shows that each node is 0.8.0
 At suggestion of some contributors, I have restarted each node and tried to 
 run a nodetool repair again ... the result is the same with the messages 
 being logged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135811 - /cassandra/trunk/conf/cassandra.yaml

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 21:43:42 2011
New Revision: 1135811

URL: http://svn.apache.org/viewvc?rev=1135811view=rev
Log:
edit cl space comments

Modified:
cassandra/trunk/conf/cassandra.yaml

Modified: cassandra/trunk/conf/cassandra.yaml
URL: 
http://svn.apache.org/viewvc/cassandra/trunk/conf/cassandra.yaml?rev=1135811r1=1135810r2=1135811view=diff
==
--- cassandra/trunk/conf/cassandra.yaml (original)
+++ cassandra/trunk/conf/cassandra.yaml Tue Jun 14 21:43:42 2011
@@ -150,8 +150,9 @@ concurrent_writes: 32
 # memtable_total_space_in_mb: 2048
 
 # Total space to use for commitlogs. 
-# if space gets above this value, Cassandra will flush every dirty CF 
-# in the oldest segment and remove it.
+# If space gets above this value (it will round up to the next nearest
+# segment multiple), Cassandra will flush every dirty CF in the oldest
+# segment and remove it.
 # commitlog_total_space_in_mb: 4096
 
 # This sets the amount of memtable flush writer threads.  These will

buildbot success in ASF Buildbot on cassandra-trunk

2011-06-14 Thread buildbot 
The Buildbot has detected a restored build on builder cassandra-trunk while 
building ASF Buildbot.
Full details are available at:
 http://ci.apache.org/builders/cassandra-trunk/builds/1365

Buildbot URL: http://ci.apache.org/

Buildslave for this Build: isis_ubuntu

Build Reason: scheduler
Build Source Stamp: [branch cassandra/trunk] 1135810
Blamelist: jbellis

Build succeeded!

sincerely,
 -The Buildbot

[jira] [Created] (CASSANDRA-2772) remove commitlog_rotation_threshold_in_mb

2011-06-14 Thread Jonathan Ellis (JIRA) 
remove commitlog_rotation_threshold_in_mb
-

 Key: CASSANDRA-2772
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2772
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jonathan Ellis
Assignee: Patricio Echague
Priority: Minor
 Fix For: 1.0


As noted in CASSANDRA-2467, nobody has found this useful in the history of the 
project. (Or the postgresql project, which also hardcodes WAL segment size.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

svn commit: r1135824 - /cassandra/branches/cassandra-0.8/CHANGES.txt

2011-06-14 Thread jbellis 
Author: jbellis
Date: Tue Jun 14 21:59:03 2011
New Revision: 1135824

URL: http://svn.apache.org/viewvc?rev=1135824view=rev
Log:
update CHANGES

Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt

Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1135824r1=1135823r2=1135824view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Tue Jun 14 21:59:03 2011
@@ -162,7 +162,8 @@
  * add key type information and alias (CASSANDRA-2311, 2396)
  * cli no longer divides read_repair_chance by 100 (CASSANDRA-2458)
  * made CompactionInfo.getTaskType return an enum (CASSANDRA-2482)
- * add a server-wide cap on measured memtable memory usage (CASSANDRA-2006)
+ * add a server-wide cap on measured memtable memory usage and aggressively
+   flush to keep under that threshold (CASSANDRA-2006)
  * add unified UUIDType (CASSANDRA-2233)

[jira] [Updated] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Vijay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay updated CASSANDRA-2491:
-

Attachment: 2491_broadcast_address_v3.patch

Rebased again

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049493#comment-13049493
 ] 

Brandon Williams commented on CASSANDRA-2491:
-

Mostly looks good.  A few things I noticed:

* If so, there's a typo in receiveMessage
* SimpleSnitch and AbstractNetworkTopologySnitch are importing DD but not using 
it
* the VersionedValue addition probably doesn't belong here

bq. Allow brodcastAddress to be set by snitch so we can automate via snitch 
(property or EC2MultiregionSnitch).

Can you explain why that's necessary?  It seems to me if the node doesn't know 
it's own BA when it starts up, it's already going to get into trouble.


 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049493#comment-13049493
 ] 

Brandon Williams edited comment on CASSANDRA-2491 at 6/14/11 10:49 PM:
---

Mostly looks good.  A few things I noticed:

* There's a typo in receiveMessage
* SimpleSnitch and AbstractNetworkTopologySnitch are importing DD but not using 
it
* the VersionedValue addition probably doesn't belong here

bq. Allow brodcastAddress to be set by snitch so we can automate via snitch 
(property or EC2MultiregionSnitch).

Can you explain why that's necessary?  It seems to me if the node doesn't know 
it's own BA when it starts up, it's already going to get into trouble.


  was (Author: brandon.williams):
Mostly looks good.  A few things I noticed:

* If so, there's a typo in receiveMessage
* SimpleSnitch and AbstractNetworkTopologySnitch are importing DD but not using 
it
* the VersionedValue addition probably doesn't belong here

bq. Allow brodcastAddress to be set by snitch so we can automate via snitch 
(property or EC2MultiregionSnitch).

Can you explain why that's necessary?  It seems to me if the node doesn't know 
it's own BA when it starts up, it's already going to get into trouble.

  
 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Vijay (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049499#comment-13049499
 ] 

Vijay commented on CASSANDRA-2491:
--

Hi Brandon,

Can you explain why that's necessary? It seems to me if the node doesn't know 
it's own BA when it starts up, it's already going to get into trouble.

-- This is Just for automation, Snitch will be initialized when 
DatabaseDescripter is initialized hence it will be safe and the Snitch is 
called only for the first time to set the BA... The fall-back if BA is not 
available then use getLocalAddress()
-- It Can also be automated outside cassandra too... let me know if you still 
think it is bad i can remove that logic.

For the rest, I will rebase it once again if you say we are good to go...

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2062) Better control of iterator consumption

2011-06-14 Thread Stu Hood (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stu Hood updated CASSANDRA-2062:


Attachment: (was: 
0001-CASSANDRA-2062-0001-Improved-iterator-for-merging-sort.txt)

 Better control of iterator consumption
 --

 Key: CASSANDRA-2062
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2062
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Stu Hood
Priority: Minor
 Fix For: 1.0


 The core reason for this ticket is to gain control over the consumption of 
 the lazy nested iterators in the read path.
 {quote}We survive now because we write the size of the row at the front of 
 the row (via some serious acrobatics at write time), which gives us hasNext() 
 for rows for free. But it became apparent while working on the block-based 
 format that hasNext() will not be cheap unless the current item has been 
 consumed. Consumption of the row is easy, and blocks will be framed so that 
 they can be very easily skipped, but you don't want to have to seek to the 
 end of the row to answer hasNext, and then seek back to the beginning to 
 consume the row, which is what CollatingIterator would have forced us to 
 do.{quote}
 While we're at it, we can also improve efficiency: for {{M}} iterators 
 containing {{N}} total items, commons.collections.CollatingIterator performs 
 a {{O(M*N)}} merge, and calls hasNext multiple times per returned value. We 
 can do better.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2062) Better control of iterator consumption

2011-06-14 Thread Stu Hood (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stu Hood updated CASSANDRA-2062:


Attachment: (was: 
0002-CASSANDRA-2062-0002-Port-all-collating-consumers-to-Me.txt)

 Better control of iterator consumption
 --

 Key: CASSANDRA-2062
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2062
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Stu Hood
Priority: Minor
 Fix For: 1.0


 The core reason for this ticket is to gain control over the consumption of 
 the lazy nested iterators in the read path.
 {quote}We survive now because we write the size of the row at the front of 
 the row (via some serious acrobatics at write time), which gives us hasNext() 
 for rows for free. But it became apparent while working on the block-based 
 format that hasNext() will not be cheap unless the current item has been 
 consumed. Consumption of the row is easy, and blocks will be framed so that 
 they can be very easily skipped, but you don't want to have to seek to the 
 end of the row to answer hasNext, and then seek back to the beginning to 
 consume the row, which is what CollatingIterator would have forced us to 
 do.{quote}
 While we're at it, we can also improve efficiency: for {{M}} iterators 
 containing {{N}} total items, commons.collections.CollatingIterator performs 
 a {{O(M*N)}} merge, and calls hasNext multiple times per returned value. We 
 can do better.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2062) Better control of iterator consumption

2011-06-14 Thread Stu Hood (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stu Hood updated CASSANDRA-2062:


Attachment: 0004-CASSANDRA-2062-0004-Replace-ReducingIterator-for-lazy-.txt
0003-CASSANDRA-2062-0003-Replace-ReducingIterator-in-QueryF.txt
0002-CASSANDRA-2062-0002-Port-all-collating-consumers-to-Me.txt
0001-CASSANDRA-2062-0001-Improved-iterator-for-merging-sort.txt

bq. [CollatingIterator] calls hasNext() on its child iterator immediately after 
pulling off the least value from one.
ReducingIterator also does not know whether there are more items until it has 
consumed past the end of the current item, which is why it was necessary to 
squash it in.

bq. we still have the RI use in collectCollatedColumns [...] as well as 
LazyColumnIterator
Fixed in 0003 and removed in 0004.

bq. It looks to me like the main obstacle to using MI there is making 
MI.Reducer support customizable isEqual?
The comparator passed to the MI is used for isEqual, so it should be pluggable 
already.


 Better control of iterator consumption
 --

 Key: CASSANDRA-2062
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2062
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Stu Hood
Priority: Minor
 Fix For: 1.0

 Attachments: 
 0001-CASSANDRA-2062-0001-Improved-iterator-for-merging-sort.txt, 
 0002-CASSANDRA-2062-0002-Port-all-collating-consumers-to-Me.txt, 
 0003-CASSANDRA-2062-0003-Replace-ReducingIterator-in-QueryF.txt, 
 0004-CASSANDRA-2062-0004-Replace-ReducingIterator-for-lazy-.txt


 The core reason for this ticket is to gain control over the consumption of 
 the lazy nested iterators in the read path.
 {quote}We survive now because we write the size of the row at the front of 
 the row (via some serious acrobatics at write time), which gives us hasNext() 
 for rows for free. But it became apparent while working on the block-based 
 format that hasNext() will not be cheap unless the current item has been 
 consumed. Consumption of the row is easy, and blocks will be framed so that 
 they can be very easily skipped, but you don't want to have to seek to the 
 end of the row to answer hasNext, and then seek back to the beginning to 
 consume the row, which is what CollatingIterator would have forced us to 
 do.{quote}
 While we're at it, we can also improve efficiency: for {{M}} iterators 
 containing {{N}} total items, commons.collections.CollatingIterator performs 
 a {{O(M*N)}} merge, and calls hasNext multiple times per returned value. We 
 can do better.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Brandon Williams (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049504#comment-13049504
 ] 

Brandon Williams commented on CASSANDRA-2491:
-

That makes sense, as long as it's doing it before gossip starts like the 
ec2snitch currently does for dc/rack.

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2771:


Attachment: CASSANDRA-2771-trunk.txt

  Remove the commitlog segment size
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2491) A new config parameter, broadcast_address

2011-06-14 Thread Vijay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay updated CASSANDRA-2491:
-

Attachment: 2491_broadcast_address_v4.patch

Thanks Brandon, Plz find the attached.

 A new config parameter, broadcast_address
 -

 Key: CASSANDRA-2491
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2491
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
 Environment: x86_64 GNU/Linux
Reporter: Khee Chin
Assignee: Khee Chin
Priority: Trivial
  Labels: patch
 Fix For: 1.0

 Attachments: 2491_broadcast_address.patch, 
 2491_broadcast_address_v2.patch, 2491_broadcast_address_v3.patch, 
 2491_broadcast_address_v4.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 A new config parameter, broadcast_address
 In a cluster setup where one or more nodes is behind a firewall and has a 
 private ip address, listen_address does not allow the hosts behind the 
 firewalls to be discovered by other nodes.
 Attached is a patch that introduces a new config parameter broadcast_address 
 which allows Cassandra nodes to explicitly specify their external ip address. 
 In addition, this allows listen_address to be set to 0.0.0.0 on the already 
 firewalled node.
 broadcast_address fallsback to listen_address when it is not stated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049520#comment-13049520
 ] 

Jonathan Ellis commented on CASSANDRA-2771:
---

can we make it private static final SEGMENT_SIZE?

  Remove the commitlog segment size
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2771:


Attachment: (was: CASSANDRA-2771-trunk.txt)

  Remove the commitlog segment size
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-2-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Patricio Echague (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049524#comment-13049524
 ] 

Patricio Echague commented on CASSANDRA-2771:
-

Fixed. Good catch.

  Remove the commitlog segment size
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-2-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2771) Remove the commitlog segment size

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2771:


Attachment: CASSANDRA-2771-2-trunk.txt

  Remove the commitlog segment size
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-2-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (CASSANDRA-2771) Remove commitlog_rotation_threshold_in_mb

2011-06-14 Thread Patricio Echague (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patricio Echague updated CASSANDRA-2771:


Summary:  Remove commitlog_rotation_threshold_in_mb  (was:  Remove the 
commitlog segment size)

  Remove commitlog_rotation_threshold_in_mb
 --

 Key: CASSANDRA-2771
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2771
 Project: Cassandra
  Issue Type: Improvement
Reporter: Patricio Echague
Assignee: Patricio Echague
Priority: Minor
  Labels: commitlog
 Fix For: 1.0

 Attachments: CASSANDRA-2771-2-trunk.txt


 Remove the commitlog segment size config setting, nobody has ever changed it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

  1   2   >