[jira] [Updated] (CASSANDRA-3242) Nodes which are not in the ring but are dead in gossip prevent truncation
[ https://issues.apache.org/jira/browse/CASSANDRA-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Harvey updated CASSANDRA-3242: Description: All of the nodes in my ring are up, however I do have an 'unreachable' node in gossip which was removed some time ago. When I attempt to run a truncation, it does not execute since some nodes are dead, despite those nodes not owning anything in the ring. Could isAnyHostDown in StorageProxy.java be modified to not care about unreachable nodes that are not in the ring? was: All of the nodes in my ring are up, however I do have an 'unreachable' node in gossip which was removed some time ago. When I attempt to run a truncation, it does to execute since some nodes are dead, despite those nodes not owning anything in the ring. Could isAnyHostDown in StorageProxy.java be modified to not care about unreachable nodes that are not in the ring? > Nodes which are not in the ring but are dead in gossip prevent truncation > - > > Key: CASSANDRA-3242 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3242 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Trivial > > All of the nodes in my ring are up, however I do have an 'unreachable' node > in gossip which was removed some time ago. When I attempt to run a > truncation, it does not execute since some nodes are dead, despite those > nodes not owning anything in the ring. > Could isAnyHostDown in StorageProxy.java be modified to not care about > unreachable nodes that are not in the ring? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3256) AssertionError when repairing a node
[
https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113908#comment-13113908
]
Jason Harvey commented on CASSANDRA-3256:
-
I should note, this same node has been repaired several times in the past with
no issues.
> AssertionError when repairing a node
>
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 0.8.5
>Reporter: Jason Harvey
>Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at
> org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at
> org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this
> assertion appears to simply check if an existing repair session could be
> found. Interestingly, the repair did continue to run after this as evidenced
> by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the
> ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3256) AssertionError when repairing a node
AssertionError when repairing a node
Key: CASSANDRA-3256
URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.5
Reporter: Jason Harvey
Priority: Minor
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at
org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at
org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. This assertion appears to be called if an
existing Repair session could not be found. Interestingly, the repair did
continue to run after this as evidenced by several other AntiEntropyService
entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring
threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3256) AssertionError when repairing a node
[
https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Harvey updated CASSANDRA-3256:
Description:
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at
org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at
org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. From peeking at the code, this assertion
appears to simply check if an existing repair session could be found.
Interestingly, the repair did continue to run after this as evidenced by
several other AntiEntropyService entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring
threw exceptions.
was:
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java
(line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at
org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at
org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. This assertion appears to be called if an
existing Repair session could not be found. Interestingly, the repair did
continue to run after this as evidenced by several other AntiEntropyService
entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring
threw exceptions.
> AssertionError when repairing a node
>
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 0.8.5
>Reporter: Jason Harvey
>Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at
> org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at
> org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at
> org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this
> assertion appears to simply check if an existing repair session could be
> found. Interestingly, the repair did continue to run after this as evidenced
> by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the
> ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[
https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113896#comment-13113896
]
Jason Harvey commented on CASSANDRA-2863:
-
Also got the same NPE when doing a repair on a different node. This time, the
NPE was preceded by a different NPE:
{code}
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:177)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:114)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382)
at
org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370)
at
org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108)
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
> NPE when writing SSTable generated via repair
> -
>
> Key: CASSANDRA-2863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2863
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 0.8.1
>Reporter: Héctor Izquierdo
> Fix For: 0.8.7
>
>
> A NPE is generated during repair when closing an sstable generated via
> SSTable build. It doesn't happen always. The node had been scrubbed and
> compacted before calling repair.
> INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line
> 158) Opening /d2/cassandra/data/sbs/walf-g-730
> ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread
> Thread[CompactionExecutor:2,1,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1103)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1094)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3255) Sstable scrub status persists in compactionstats after scrub is complete
Sstable scrub status persists in compactionstats after scrub is complete Key: CASSANDRA-3255 URL: https://issues.apache.org/jira/browse/CASSANDRA-3255 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.5 Reporter: Jason Harvey Priority: Trivial When scrubbing the sstables on a node, the 'Scrub' info persists in the 'compactionstats' nodetool utility, even after the scrub is complete. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3254) MessagingService sends blank bytes
[ https://issues.apache.org/jira/browse/CASSANDRA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hinkle updated CASSANDRA-3254: --- Attachment: cassandra-0.8-3254-v2.txt V2 removes the extra copy. I wasn't sure if you wanted the additional code change from your message, but this is what I was testing with. I also wasn't sure if packIt was considered an API since it is public (but not used anywhere else in the codebase). I tested a n=3,rf=3 cluster with a portion of our app that sends large batches of counter mutations (mostly in separate rows). The separate replicate on write messages per row are where this has the biggest impact for us. The small test increments 1M counters in separate rows. The fix reduces total network traffic by about 360MB or 30% of 1,148MB (as measured at the network interfaces). It is roughly 10% faster. Though the network obviously makes the timings vary a fair amount, the network traffic numbers are consistent. > MessagingService sends blank bytes > -- > > Key: CASSANDRA-3254 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3254 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.6 >Reporter: Greg Hinkle > Attachments: cassandra-0.8-3254-v2.txt, cassandra-0.8-3254.txt > > > MessagingService is calling DataOutputBuffer.getData() instead of > .asByteArray() in two places. This results in always sending the full buffer > size of at least 128 bytes even if the message is smaller. For messages like > gossip and single col mutations its around 40 to 80 bytes wasted per message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3020) Failures in system test: test_cql.py
[
https://issues.apache.org/jira/browse/CASSANDRA-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cathy Daw resolved CASSANDRA-3020.
--
Resolution: Not A Problem
Assignee: Cathy Daw (was: Tyler Hobbs)
issue with driver package moving
> Failures in system test: test_cql.py
>
>
> Key: CASSANDRA-3020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3020
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 0.8.5
> Environment: 0.8 branch for Aug 11th test run:
> https://jenkins.qa.datastax.com/job/CassandraSystem/142/
> https://jenkins.qa.datastax.com/job/Cassandra/147/
>Reporter: Cathy Daw
>Assignee: Cathy Daw
>
> *Test Output*
> {code}
> ==
> ERROR: reading and writing strings w/ newlines
> --
> Traceback (most recent call last):
> File "/usr/local/lib/python2.6/dist-packages/nose/case.py", line 187, in
> runTest
> self.test(*self.arg)
> File "/var/lib/jenkins/jobs/Cassandra/workspace/test/system/test_cql.py",
> line 734, in test_newline_strings
> """, {"key": "\nkey", "name": "\nname"})
> File "/var/lib/jenkins/repos/drivers/py/cql/cursor.py", line 150, in execute
> self.description = self.decoder.decode_description(self._query_ks,
> self._query_cf, self.result[0])
> File "/var/lib/jenkins/repos/drivers/py/cql/decoders.py", line 39, in
> decode_description
> comparator = self.__comparator_for(keyspace, column_family)
> File "/var/lib/jenkins/repos/drivers/py/cql/decoders.py", line 35, in
> __comparator_for
> return cfam.get("comparator", None)
> AttributeError: 'NoneType' object has no attribute 'get'
> --
> Ran 127 tests in 635.426s
> FAILED (errors=1)
> Sending e-mails to: q...@datastax.com
> Finished: FAILURE
> {code}
> *Suspected check-in*
> {code}
> Revision 1156198 by xedin:
> Fixes issues with parameters being escaped incorrectly in Python CQL
> patch by Tyler Hobbs; reviewed by Pavel Yaskevich for CASSANDRA-2993
> /cassandra/branches/cassandra-0.8/test/system/test_cql.py
> /cassandra/branches/cassandra-0.8/CHANGES.txt
> /cassandra/drivers/py/cql/cursor.py
>
> /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cql/Cql.g
> /cassandra/drivers/py/test/test_regex.py
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3243: Fix Version/s: 0.8.7 > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Fix For: 0.8.7 > > Attachments: 3243.txt, locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3243: Attachment: 3243.txt The best explanation I have for how you could get here is that SystemTable only forces a flush on the updateToken(Token token) signature. removeToken and updateToken(InetAddress ep, Token token) do not, so if the machine is restarted before the commitlog is synced, the update/removal can be lost. Patch to address this. > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Attachments: 3243.txt, locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113819#comment-13113819 ] Jason Harvey commented on CASSANDRA-3243: - For outside reference: Brandon recommended I delete the 'Ring' key from the LocationInfo on this node and then restart to resolve the weirdness. > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Attachments: locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3206) increase file descriptor limit in deb, rpm packages
[ https://issues.apache.org/jira/browse/CASSANDRA-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113816#comment-13113816 ] Hudson commented on CASSANDRA-3206: --- Integrated in Cassandra-0.8 #339 (See [https://builds.apache.org/job/Cassandra-0.8/339/]) Increase FD limit to 100k in packaging. Patch by Paul Cannon, reviewed by brandonwilliams for CASSANDRA-3206 brandonwilliams : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175027 Files : * /cassandra/branches/cassandra-0.8/CHANGES.txt * /cassandra/branches/cassandra-0.8/debian/cassandra.conf * /cassandra/branches/cassandra-0.8/debian/init * /cassandra/branches/cassandra-0.8/redhat/cassandra.conf > increase file descriptor limit in deb, rpm packages > --- > > Key: CASSANDRA-3206 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3206 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Jonathan Ellis >Assignee: paul cannon >Priority: Minor > Fix For: 0.8.7 > > Attachments: 3206.patch.txt > > > We can use a lot of file descriptors (one per socket, 5? per sstable). > People hit this regularly on the user list and it will get worse with Leveled > compaction, which limits sstable size to a relatively low size (currently > 5MB). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1175058 - in /cassandra/branches/cassandra-1.0.0: ./ contrib/ debian/ interface/thrift/gen-java/org/apache/cassandra/thrift/ redhat/
Author: jbellis Date: Fri Sep 23 22:31:36 2011 New Revision: 1175058 URL: http://svn.apache.org/viewvc?rev=1175058&view=rev Log: merge from 0.8 Modified: cassandra/branches/cassandra-1.0.0/ (props changed) cassandra/branches/cassandra-1.0.0/CHANGES.txt cassandra/branches/cassandra-1.0.0/NEWS.txt cassandra/branches/cassandra-1.0.0/contrib/ (props changed) cassandra/branches/cassandra-1.0.0/debian/cassandra.conf cassandra/branches/cassandra-1.0.0/debian/init cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-1.0.0/redhat/cassandra.conf Propchange: cassandra/branches/cassandra-1.0.0/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 22:31:36 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469,1174701 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1175057 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 Modified: cassandra/branches/cassandra-1.0.0/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/CHANGES.txt?rev=1175058&r1=1175057&r2=1175058&view=diff == --- cassandra/branches/cassandra-1.0.0/CHANGES.txt (original) +++ cassandra/branches/cassandra-1.0.0/CHANGES.txt Fri Sep 23 22:31:36 2011 @@ -17,6 +17,7 @@ * Allow using quotes in "USE ;" CLI command (CASSANDRA-3208) * Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218) * Fix sstableloader --ignores option (CASSANDRA-3247) + * File descriptor limit increased in packaging (CASSANDRA-3206) 1.0.0-beta1 Modified: cassandra/branches/cassandra-1.0.0/NEWS.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/NEWS.txt?rev=1175058&r1=1175057&r2=1175058&view=diff == --- cassandra/branches/cassandra-1.0.0/NEWS.txt (original) +++ cassandra/branches/cassandra-1.0.0/NEWS.txt Fri Sep 23 22:31:36 2011 @@ -204,7 +204,7 @@ Features - optional intranode encryption; see comments around 'encryption_options' in cassandra.yaml - compaction multithreading and rate-limiting; see - 'compaction_multithreading' and 'compaction_throughput_mb_per_sec' in + 'concurrent_compactors' and 'compaction_throughput_mb_per_sec' in cassandra.yaml - cassandra will limit total memtable memory usage to 1/3 of the heap by default. This can be ajusted or disabled with the Propchange: cassandra/branches/cassandra-1.0.0/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 22:31:36 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469,1174701 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1175057 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 Modified: cassandra/branches/cassandra-1.0.0/debian/cassandra.conf URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/debian/cassandra.conf?rev=1175058&r1=1175057&r2=1175058&view=diff == --- cassandra/branches/cassandra-1.0.0/debian/cassandra.conf (original) +++ cassandra/branches/cassandra-1.0.0/debian/cassandra.conf Fri Sep 23 22:31:36 2011 @@ -1,2 +1,3 @@ # Provided by the cassandra package cassandra - memlock unlimited +cassandra - nofile 10 Modified: cassandra/branches/cassandra-1.0.0/debian/init URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0
[jira] [Issue Comment Edited] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113812#comment-13113812 ] Jason Harvey edited comment on CASSANDRA-3243 at 9/23/11 10:30 PM: --- Clock on these boxes seems fine. We keep ntpd running at all times. I've also verified via logging that it has been consistent. How do I go about getting that endpoint *out* of the LocationInfo? was (Author: alienth): Clock on these boxes seems fine. We keep ntpd running at all times. I've also verified via logging that it has been consistent. > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Attachments: locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1175057 - /cassandra/branches/cassandra-0.8/NEWS.txt
Author: jbellis Date: Fri Sep 23 22:30:59 2011 New Revision: 1175057 URL: http://svn.apache.org/viewvc?rev=1175057&view=rev Log: fix NEWS for #2191 Modified: cassandra/branches/cassandra-0.8/NEWS.txt Modified: cassandra/branches/cassandra-0.8/NEWS.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/NEWS.txt?rev=1175057&r1=1175056&r2=1175057&view=diff == --- cassandra/branches/cassandra-0.8/NEWS.txt (original) +++ cassandra/branches/cassandra-0.8/NEWS.txt Fri Sep 23 22:30:59 2011 @@ -147,7 +147,7 @@ Features - optional intranode encryption; see comments around 'encryption_options' in cassandra.yaml - compaction multithreading and rate-limiting; see - 'compaction_multithreading' and 'compaction_throughput_mb_per_sec' in + 'concurrent_compactors' and 'compaction_throughput_mb_per_sec' in cassandra.yaml - cassandra will limit total memtable memory usage to 1/3 of the heap by default. This can be ajusted or disabled with the
[jira] [Commented] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113812#comment-13113812 ] Jason Harvey commented on CASSANDRA-3243: - Clock on these boxes seems fine. We keep ntpd running at all times. I've also verified via logging that it has been consistent. > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Attachments: locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3254) MessagingService sends blank bytes
[ https://issues.apache.org/jira/browse/CASSANDRA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113811#comment-13113811 ] Jonathan Ellis commented on CASSANDRA-3254: --- (Especially since for larger mutations the space saved is not as significant but you are stuck with the extra copies.) > MessagingService sends blank bytes > -- > > Key: CASSANDRA-3254 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3254 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.6 >Reporter: Greg Hinkle > Attachments: cassandra-0.8-3254.txt > > > MessagingService is calling DataOutputBuffer.getData() instead of > .asByteArray() in two places. This results in always sending the full buffer > size of at least 128 bytes even if the message is smaller. For messages like > gossip and single col mutations its around 40 to 80 bytes wasted per message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3254) MessagingService sends blank bytes
[ https://issues.apache.org/jira/browse/CASSANDRA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113810#comment-13113810 ] Jonathan Ellis commented on CASSANDRA-3254: --- Do you see a performance difference in practice? The problem is that this cuts down the extra bytes by adding copies to extra byte[], so I'm not sure it's a big win without avoiding the copies too a la CASSANDRA-1788. > MessagingService sends blank bytes > -- > > Key: CASSANDRA-3254 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3254 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.6 >Reporter: Greg Hinkle > Attachments: cassandra-0.8-3254.txt > > > MessagingService is calling DataOutputBuffer.getData() instead of > .asByteArray() in two places. This results in always sending the full buffer > size of at least 128 bytes even if the message is smaller. For messages like > gossip and single col mutations its around 40 to 80 bytes wasted per message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1175027 - in /cassandra/branches/cassandra-0.8: CHANGES.txt debian/cassandra.conf debian/init redhat/cassandra.conf
Author: brandonwilliams
Date: Fri Sep 23 21:23:37 2011
New Revision: 1175027
URL: http://svn.apache.org/viewvc?rev=1175027&view=rev
Log:
Increase FD limit to 100k in packaging.
Patch by Paul Cannon, reviewed by brandonwilliams for CASSANDRA-3206
Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt
cassandra/branches/cassandra-0.8/debian/cassandra.conf
cassandra/branches/cassandra-0.8/debian/init
cassandra/branches/cassandra-0.8/redhat/cassandra.conf
Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1175027&r1=1175026&r2=1175027&view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Fri Sep 23 21:23:37 2011
@@ -4,6 +4,7 @@
* Log message when a full repair operation completes (CASSANDRA-3207)
* Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218)
* Fix sstableloader --ignores option (CASSANDRA-3247)
+ * File descriptor limit increased in packaging (CASSANDRA-3206)
0.8.6
Modified: cassandra/branches/cassandra-0.8/debian/cassandra.conf
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/debian/cassandra.conf?rev=1175027&r1=1175026&r2=1175027&view=diff
==
--- cassandra/branches/cassandra-0.8/debian/cassandra.conf (original)
+++ cassandra/branches/cassandra-0.8/debian/cassandra.conf Fri Sep 23 21:23:37
2011
@@ -1,2 +1,3 @@
# Provided by the cassandra package
cassandra - memlock unlimited
+cassandra - nofile 10
Modified: cassandra/branches/cassandra-0.8/debian/init
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/debian/init?rev=1175027&r1=1175026&r2=1175027&view=diff
==
--- cassandra/branches/cassandra-0.8/debian/init (original)
+++ cassandra/branches/cassandra-0.8/debian/init Fri Sep 23 21:23:37 2011
@@ -22,6 +22,7 @@ CONFDIR=/etc/cassandra
JSVC=/usr/bin/jsvc
WAIT_FOR_START=10
CASSANDRA_HOME=/usr/share/cassandra
+FD_LIMIT=10
# The first existing directory is used for JAVA_HOME if needed.
JVM_SEARCH_DIRS="/usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-sun"
@@ -127,6 +128,7 @@ do_start()
is_running && return 1
ulimit -l unlimited
+ulimit -n "$FD_LIMIT"
cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6; }'`
cd /# jsvc doesn't chdir() for us
Modified: cassandra/branches/cassandra-0.8/redhat/cassandra.conf
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/redhat/cassandra.conf?rev=1175027&r1=1175026&r2=1175027&view=diff
==
--- cassandra/branches/cassandra-0.8/redhat/cassandra.conf (original)
+++ cassandra/branches/cassandra-0.8/redhat/cassandra.conf Fri Sep 23 21:23:37
2011
@@ -1 +1,2 @@
cassandra - memlock unlimited
+cassandra - nofile 10
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113778#comment-13113778 ] paul cannon commented on CASSANDRA-2961: +1 > Expire dead gossip states based on time > --- > > Key: CASSANDRA-2961 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 > Project: Cassandra > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Brandon Williams >Assignee: Jérémy Sevellec >Priority: Minor > Fix For: 1.0.1 > > Attachments: trunk-2961-v2.patch, trunk-2961-v3.patch, > trunk-2961-v4.patch, trunk-2961.patch > > > Currently dead states are held until aVeryLongTime, 3 days. The problem is > that if a node reboots within this period, it begins a new 3 days and will > repopulate the ring with the dead state. While mostly harmless, perpetuating > the state forever is at least wasting a small amount of bandwidth. Instead, > we can expire states based on a ttl, which will require that the cluster be > loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-2863: Fix Version/s: 0.8.7 > NPE when writing SSTable generated via repair > - > > Key: CASSANDRA-2863 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2863 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.1 >Reporter: Héctor Izquierdo > Fix For: 0.8.7 > > > A NPE is generated during repair when closing an sstable generated via > SSTable build. It doesn't happen always. The node had been scrubbed and > compacted before calling repair. > INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line > 158) Opening /d2/cassandra/data/sbs/walf-g-730 > ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:2,1,main] > java.lang.NullPointerException > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1103) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1094) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams reopened CASSANDRA-2863: - > NPE when writing SSTable generated via repair > - > > Key: CASSANDRA-2863 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2863 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.1 >Reporter: Héctor Izquierdo > Fix For: 0.8.7 > > > A NPE is generated during repair when closing an sstable generated via > SSTable build. It doesn't happen always. The node had been scrubbed and > compacted before calling repair. > INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line > 158) Opening /d2/cassandra/data/sbs/walf-g-730 > ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:2,1,main] > java.lang.NullPointerException > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1103) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1094) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3206) increase file descriptor limit in deb, rpm packages
[ https://issues.apache.org/jira/browse/CASSANDRA-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] paul cannon updated CASSANDRA-3206: --- Attachment: 3206.patch.txt > increase file descriptor limit in deb, rpm packages > --- > > Key: CASSANDRA-3206 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3206 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Jonathan Ellis >Assignee: paul cannon >Priority: Minor > Fix For: 0.8.7 > > Attachments: 3206.patch.txt > > > We can use a lot of file descriptors (one per socket, 5? per sstable). > People hit this regularly on the user list and it will get worse with Leveled > compaction, which limits sstable size to a relatively low size (currently > 5MB). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[
https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113771#comment-13113771
]
Jason Harvey commented on CASSANDRA-2863:
-
Getting the following after trying to 'move' on 0.8.5:
{code}
ERROR [CompactionExecutor:213] 2011-09-23 14:02:26,571
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[CompactionExecutor:213,1,main]
java.lang.NullPointerException
at
org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382)
at
org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370)
at
org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108)
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
> NPE when writing SSTable generated via repair
> -
>
> Key: CASSANDRA-2863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2863
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 0.8.1
>Reporter: Héctor Izquierdo
>
> A NPE is generated during repair when closing an sstable generated via
> SSTable build. It doesn't happen always. The node had been scrubbed and
> compacted before calling repair.
> INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line
> 158) Opening /d2/cassandra/data/sbs/walf-g-730
> ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread
> Thread[CompactionExecutor:2,1,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1103)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1094)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1991) CFS.maybeSwitchMemtable() calls CommitLog.instance.getContext(), which may block, under flusher lock write lock
[
https://issues.apache.org/jira/browse/CASSANDRA-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113756#comment-13113756
]
Yang Yang commented on CASSANDRA-1991:
--
any updates on this JIRA ? the code concerned here causes a more serious issue
in 3253, so I'd like to see a unified solution for both.
thanks
yang
> CFS.maybeSwitchMemtable() calls CommitLog.instance.getContext(), which may
> block, under flusher lock write lock
> ---
>
> Key: CASSANDRA-1991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1991
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Peter Schuller
>Assignee: Peter Schuller
> Attachments: 1991-checkpointing-flush.txt, 1991-logchanges.txt,
> 1991-trunk-v2.txt, 1991-trunk.txt, 1991-v3.txt, 1991-v4.txt, 1991-v5.txt,
> 1991-v6.txt, 1991-v7.txt, 1991-v8.txt, 1991-v9.txt, trigger.py
>
>
> While investigate CASSANDRA-1955 I realized I was seeing very poor latencies
> for reasons that had nothing to do with flush_writers, even when using
> periodic commit log mode (and flush writers set ridiculously high, 500).
> It turns out writes blocked were slow because Table.apply() was spending lots
> of time (I can easily trigger seconds on moderate work-load) trying to
> acquire a flusher lock read lock ("flush lock millis" log printout in the
> logging patch I'll attach).
> That in turns is caused by CFS.maybeSwitchMemtable() which acquires the
> flusher lock write lock.
> Bisecting further revealed that the offending line of code that blocked was:
> final CommitLogSegment.CommitLogContext ctx =
> writeCommitLog ? CommitLog.instance.getContext() : null;
> Indeed, CommitLog.getContext() simply returns currentSegment().getContext(),
> but does so by submitting a callable on the service executor. So
> independently of flush writers, this can block all (global, for all cf:s)
> writes very easily, and does.
> I'll attach a file that is an independent Python script that triggers it on
> my macos laptop (with an intel SSD, which is why I was particularly
> surprised) (it assumes CPython, out-of-the-box-or-almost Cassandra on
> localhost that isn't in a cluster, and it will drop/recreate a keyspace
> called '1955').
> I'm also attaching, just FYI, the patch with log entries that I used while
> tracking it down.
> Finally, I'll attach a patch with a suggested solution of keeping track of
> the latest commit log with an AtomicReference (as an alternative to
> synchronizing all access to segments). With that patch applied, latencies are
> not affected by my trigger case like they were before. There are some
> sub-optimal > 100 ms cases on my test machine, but for other reasons. I'm no
> longer able to trigger the extremes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3243) Node which was decommissioned and shut-down reappears on a single node
[ https://issues.apache.org/jira/browse/CASSANDRA-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113713#comment-13113713 ] Brandon Williams commented on CASSANDRA-3243: - 0919 is missing the LOCATION_KEY (the node's own token) which is odd, because cassandra will refuse to startup with this table since it should not exist without this key. It does show itself in the saved endpoints, but no other nodes. 0922 is complete in that it contains LOCATION_KEY and cassandra starts right up with it, and I can see the removed token in the saved endpoints with an ip address of 10.34.22.201. However the strange thing is the timestamp on that column is approximately 2 days _older_ than the one for the local node itself, which should be impossible. Is there any chance this node's clock was way off or changed? > Node which was decommissioned and shut-down reappears on a single node > -- > > Key: CASSANDRA-3243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3243 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.5 >Reporter: Jason Harvey >Assignee: Brandon Williams >Priority: Minor > Attachments: locationinfo_0919.tgz, locationinfo_0922.tgz > > > I decommissioned a node several days ago. It was no longer in the ring list > on any node in the ring. However, it was in the dead gossip list. > In an attempt to clean it out of the dead gossip list so I could truncate, I > shut down the entire ring and bought it back up. Once the ring came back up, > one node showed the decommissioned node as still in the ring in a state of > 'Down'. No other node in the ring shows this info. > I successfully ran removetoken on the node to get that phantom node out. > However, it is back in the dead gossip list, preventing me from truncating. > Where might the info on this decommissioned node be being stored? Is HH > possibly trying to deliver to the removed node, thus putting it back in the > ring on one node? > I find it extremely curious that none of the other nodes in the ring showed > the phantom node. Shouldn't gossip have propagated the node everywhere, even > if it was down? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3238) Issue with multi region ec2 and replication updates
[ https://issues.apache.org/jira/browse/CASSANDRA-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113687#comment-13113687 ] Vijay commented on CASSANDRA-3238: -- Talked with the user and it seems like we will have to set dynamic_snitch_badness_threshold=0.01 atleast. so far with that setting the problem seem to go away. > Issue with multi region ec2 and replication updates > --- > > Key: CASSANDRA-3238 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3238 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Nick Bailey >Assignee: Vijay >Priority: Minor > Fix For: 1.0.0 > > > Using the Ec2MultiRegionSnitch and updating replication settings for a > keyspace seems to cause some issues that require a rolling restart to fix. > The following was observed when updating a keyspace from SimpleStrategy to > NTS in a multi region environment: > * All repairs would hang. Even repairs only against a keyspace that was not > updated. > * Reads at CL.ONE would start to go across region > After a rolling restart of the cluster, repairs started working correctly > again and reads stayed local to the region. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3252) IntegerType columns cause exceptions when use in Pig.
[ https://issues.apache.org/jira/browse/CASSANDRA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113682#comment-13113682 ] Jonathan Ellis commented on CASSANDRA-3252: --- int32type is new in 1.0 > IntegerType columns cause exceptions when use in Pig. > - > > Key: CASSANDRA-3252 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3252 > Project: Cassandra > Issue Type: Bug > Components: Contrib, Hadoop >Affects Versions: 0.8.6 >Reporter: Dana H. P'Simer, Jr. > Labels: cassandra, hadoop, pig > Attachments: CASSANDRA-3252.patch > > > CassandraStorage uses the validator classes to determine how to marshal > column values, if the type is IntegerType a BigInteger is returned and Pig > throws an exception on writing the value to intermediate storage. The > exception is thrown because BigInteger is not one of the data types that Pig > can handle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326
]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 7:30 PM:
---
This is persisting to be a problem. And continuously gets worse through the day.
I can see two thirds of my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner. Although
i'm having a greater problem with the former. A compact and restart didn't
help.
I've attached two screenshots from hadoop webpages (you see 36million map input
records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite
how terribly long they may take) that the results are correct. It would seem
that the split ranges are incorrect (not the rows within in them).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the
day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input
records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite
how terribly long they may take) that the results are correct. It would seem
that the split ranges are incorrect (not the rows within in them).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for
> task_201109212019_1060_m_29 - Mozilla Firefox.png, Screenshot-Hadoop map
> task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3253) inherent deadlock situation in commitLog flush?
[ https://issues.apache.org/jira/browse/CASSANDRA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113655#comment-13113655 ] Yang Yang edited comment on CASSANDRA-3253 at 9/23/11 7:27 PM: --- looks to be related to https://issues.apache.org/jira/browse/CASSANDRA-1991 was (Author: yangyangyyy): looks to be the same of https://issues.apache.org/jira/browse/CASSANDRA-1991 > inherent deadlock situation in commitLog flush? > --- > > Key: CASSANDRA-3253 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3253 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Yang Yang > > after my system ran for a while, it consitently goes into frozen state where > all the mutations stage threads are waiting > on the switchlock, > the reason is that the switchlock is held by commit log, as shown by the > following thread dump: > "COMMIT-LOG-WRITER" prio=10 tid=0x010df000 nid=0x32d3 waiting on > condition [0x7f2d81557000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7f3579eec060> (a > java.util.concurrent.FutureTask$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:386) > at > org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:650) > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:722) > at > org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:573) > at > org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:81) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:596) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:679) > we can clearly see that the COMMIT-LOG-WRITER thread is running the regular > appender , but the appender itself calls getContext(), which again submits a > new Callable to be executed, and waits on the Callable. but the new Callable > is never going to be executed since the executor has only *one* thread. > I believe this is a deterministic bug. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3254) MessagingService sends blank bytes
[ https://issues.apache.org/jira/browse/CASSANDRA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hinkle updated CASSANDRA-3254: --- Attachment: cassandra-0.8-3254.txt Simplest fix > MessagingService sends blank bytes > -- > > Key: CASSANDRA-3254 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3254 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.6 >Reporter: Greg Hinkle > Attachments: cassandra-0.8-3254.txt > > > MessagingService is calling DataOutputBuffer.getData() instead of > .asByteArray() in two places. This results in always sending the full buffer > size of at least 128 bytes even if the message is smaller. For messages like > gossip and single col mutations its around 40 to 80 bytes wasted per message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3254) MessagingService sends blank bytes
MessagingService sends blank bytes -- Key: CASSANDRA-3254 URL: https://issues.apache.org/jira/browse/CASSANDRA-3254 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.6 Reporter: Greg Hinkle MessagingService is calling DataOutputBuffer.getData() instead of .asByteArray() in two places. This results in always sending the full buffer size of at least 128 bytes even if the message is smaller. For messages like gossip and single col mutations its around 40 to 80 bytes wasted per message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3252) IntegerType columns cause exceptions when use in Pig.
[ https://issues.apache.org/jira/browse/CASSANDRA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113656#comment-13113656 ] Dana H. P'Simer, Jr. commented on CASSANDRA-3252: - @Jonathan, I did not see any int32type and I just checked again and there is no such type in the org.apache.cassandra.db.marshal package. Where is this type? On the other hand, LongType is not appropriate since it expectes 8 bytes and there are only 4 in the data. @Brandon, even the latest version of that patch does not handle the case where the BigInteger is larger than 32 bits. I assume that BigInteger was used to allow for arbitrarily large integers. Your code would break if the data were larger than 32 bits. However, I do like moving the logic into a separate function. > IntegerType columns cause exceptions when use in Pig. > - > > Key: CASSANDRA-3252 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3252 > Project: Cassandra > Issue Type: Bug > Components: Contrib, Hadoop >Affects Versions: 0.8.6 >Reporter: Dana H. P'Simer, Jr. > Labels: cassandra, hadoop, pig > Attachments: CASSANDRA-3252.patch > > > CassandraStorage uses the validator classes to determine how to marshal > column values, if the type is IntegerType a BigInteger is returned and Pig > throws an exception on writing the value to intermediate storage. The > exception is thrown because BigInteger is not one of the data types that Pig > can handle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3253) inherent deadlock situation in commitLog flush?
[ https://issues.apache.org/jira/browse/CASSANDRA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113655#comment-13113655 ] Yang Yang commented on CASSANDRA-3253: -- looks to be the same of https://issues.apache.org/jira/browse/CASSANDRA-1991 > inherent deadlock situation in commitLog flush? > --- > > Key: CASSANDRA-3253 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3253 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Yang Yang > > after my system ran for a while, it consitently goes into frozen state where > all the mutations stage threads are waiting > on the switchlock, > the reason is that the switchlock is held by commit log, as shown by the > following thread dump: > "COMMIT-LOG-WRITER" prio=10 tid=0x010df000 nid=0x32d3 waiting on > condition [0x7f2d81557000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7f3579eec060> (a > java.util.concurrent.FutureTask$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:386) > at > org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:650) > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:722) > at > org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:573) > at > org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:81) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:596) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:679) > we can clearly see that the COMMIT-LOG-WRITER thread is running the regular > appender , but the appender itself calls getContext(), which again submits a > new Callable to be executed, and waits on the Callable. but the new Callable > is never going to be executed since the executor has only *one* thread. > I believe this is a deterministic bug. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3253) inherent deadlock situation in commitLog flush?
[ https://issues.apache.org/jira/browse/CASSANDRA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113621#comment-13113621 ] Yang Yang commented on CASSANDRA-3253: -- it seems to be added in the recent feature to flush earliest segment with dirty CFs : https://github.com/apache/cassandra/commit/f599559221ad074d9af0a99d7ffdd482c2b6b10c#diff-3 CFS.forceFlush() was added to the commit log writing path > inherent deadlock situation in commitLog flush? > --- > > Key: CASSANDRA-3253 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3253 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Yang Yang > > after my system ran for a while, it consitently goes into frozen state where > all the mutations stage threads are waiting > on the switchlock, > the reason is that the switchlock is held by commit log, as shown by the > following thread dump: > "COMMIT-LOG-WRITER" prio=10 tid=0x010df000 nid=0x32d3 waiting on > condition [0x7f2d81557000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7f3579eec060> (a > java.util.concurrent.FutureTask$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:386) > at > org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:650) > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:722) > at > org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:573) > at > org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:81) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:596) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:679) > we can clearly see that the COMMIT-LOG-WRITER thread is running the regular > appender , but the appender itself calls getContext(), which again submits a > new Callable to be executed, and waits on the Callable. but the new Callable > is never going to be executed since the executor has only *one* thread. > I believe this is a deterministic bug. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3252) IntegerType columns cause exceptions when use in Pig.
[ https://issues.apache.org/jira/browse/CASSANDRA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113619#comment-13113619 ] Jonathan Ellis commented on CASSANDRA-3252: --- Wouldn't a better fix be "use longtype or int32type if that's what you want instead?" > IntegerType columns cause exceptions when use in Pig. > - > > Key: CASSANDRA-3252 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3252 > Project: Cassandra > Issue Type: Bug > Components: Contrib, Hadoop >Affects Versions: 0.8.6 >Reporter: Dana H. P'Simer, Jr. > Labels: cassandra, hadoop, pig > Attachments: CASSANDRA-3252.patch > > > CassandraStorage uses the validator classes to determine how to marshal > column values, if the type is IntegerType a BigInteger is returned and Pig > throws an exception on writing the value to intermediate storage. The > exception is thrown because BigInteger is not one of the data types that Pig > can handle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3214) Make CFIF use rpc_endpoint prior to trying endpoint
[ https://issues.apache.org/jira/browse/CASSANDRA-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113613#comment-13113613 ] Jonathan Ellis commented on CASSANDRA-3214: --- But it's totally valid to have rpc_ on 0.0.0.0 and listen on something that clients cannot connect to (separate interface, firewalled off, etc). So fundamentally I think it comes down to "you shouldn't use 0.0.0.0 as rpc_address if you're going to use describe_ring." > Make CFIF use rpc_endpoint prior to trying endpoint > --- > > Key: CASSANDRA-3214 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3214 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Affects Versions: 0.8.6 > Environment: Hadoop 0.20 / Cassandra 0.8.5 / Ubuntu 10.04 / >Reporter: Eldon Stegall >Assignee: Eldon Stegall >Priority: Minor > Fix For: 0.8.7, 1.0.0 > > Attachments: CASSANDRA-3214-make-cfif-use-rpc-endpoints-v1.txt > > > Following up on CASSANDRA-3187 , we probably need to attempt to use the > rpc_endpoint address first, and then fall back to the gossip endpoint if we > don't get what we want. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3253) inherent deadlock situation in commitLog flush?
inherent deadlock situation in commitLog flush? --- Key: CASSANDRA-3253 URL: https://issues.apache.org/jira/browse/CASSANDRA-3253 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Reporter: Yang Yang after my system ran for a while, it consitently goes into frozen state where all the mutations stage threads are waiting on the switchlock, the reason is that the switchlock is held by commit log, as shown by the following thread dump: "COMMIT-LOG-WRITER" prio=10 tid=0x010df000 nid=0x32d3 waiting on condition [0x7f2d81557000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x7f3579eec060> (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:386) at org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:650) at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:722) at org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:573) at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:81) at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:596) at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:679) we can clearly see that the COMMIT-LOG-WRITER thread is running the regular appender , but the appender itself calls getContext(), which again submits a new Callable to be executed, and waits on the Callable. but the new Callable is never going to be executed since the executor has only *one* thread. I believe this is a deterministic bug. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3252) IntegerType columns cause exceptions when use in Pig.
[ https://issues.apache.org/jira/browse/CASSANDRA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dana H. P'Simer, Jr. updated CASSANDRA-3252: Attachment: CASSANDRA-3252.patch A somewhat kludgey fix for IntegerType data in cassandra. > IntegerType columns cause exceptions when use in Pig. > - > > Key: CASSANDRA-3252 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3252 > Project: Cassandra > Issue Type: Bug > Components: Contrib, Hadoop >Affects Versions: 0.8.6 >Reporter: Dana H. P'Simer, Jr. > Labels: cassandra, hadoop, pig > Attachments: CASSANDRA-3252.patch > > > CassandraStorage uses the validator classes to determine how to marshal > column values, if the type is IntegerType a BigInteger is returned and Pig > throws an exception on writing the value to intermediate storage. The > exception is thrown because BigInteger is not one of the data types that Pig > can handle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3252) IntegerType columns cause exceptions when use in Pig.
IntegerType columns cause exceptions when use in Pig. - Key: CASSANDRA-3252 URL: https://issues.apache.org/jira/browse/CASSANDRA-3252 Project: Cassandra Issue Type: Bug Components: Contrib, Hadoop Affects Versions: 0.8.6 Reporter: Dana H. P'Simer, Jr. CassandraStorage uses the validator classes to determine how to marshal column values, if the type is IntegerType a BigInteger is returned and Pig throws an exception on writing the value to intermediate storage. The exception is thrown because BigInteger is not one of the data types that Pig can handle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3140) Expose server, api versions to CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish babu krishnamoorthy updated CASSANDRA-3140: -- Attachment: 3140-patch1.diff Added support for SELECT APIVERSION and SELECT SERVERVERSION to CQL > Expose server, api versions to CQL > -- > > Key: CASSANDRA-3140 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3140 > Project: Cassandra > Issue Type: New Feature >Reporter: Jonathan Ellis >Assignee: satish babu krishnamoorthy >Priority: Minor > Fix For: 1.0.1 > > Attachments: 3140-patch.diff, 3140-patch1.diff > > > Need to expose the CQL api version; might as well include the server version > while we're at it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3251) CassandraStorage uses comparator for both super column names and sub column names.
[ https://issues.apache.org/jira/browse/CASSANDRA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dana H. P'Simer, Jr. updated CASSANDRA-3251: Attachment: CASSANDRA-3251.patch > CassandraStorage uses comparator for both super column names and sub column > names. > -- > > Key: CASSANDRA-3251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3251 > Project: Cassandra > Issue Type: Bug > Components: Contrib, Hadoop >Affects Versions: 0.8.6 >Reporter: Dana H. P'Simer, Jr. > Labels: cassandra, hadoop, pig > Attachments: CASSANDRA-3251.patch > > > The CassandraStorage class uses the same comparator for super and sub column > names. > This is because it calls columnsToTuple recursively without any indication > that the subsequent call is for sub columns. Also, the getDefaultMarshallers > method does not return the sub column name comparator. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1174871 - in /cassandra/site: publish/download/index.html src/content/download/index.html
Author: slebresne
Date: Fri Sep 23 16:27:51 2011
New Revision: 1174871
URL: http://svn.apache.org/viewvc?rev=1174871&view=rev
Log:
Fix SHA1 link on website
Modified:
cassandra/site/publish/download/index.html
cassandra/site/src/content/download/index.html
Modified: cassandra/site/publish/download/index.html
URL:
http://svn.apache.org/viewvc/cassandra/site/publish/download/index.html?rev=1174871&r1=1174870&r2=1174871&view=diff
==
--- cassandra/site/publish/download/index.html (original)
+++ cassandra/site/publish/download/index.html Fri Sep 23 16:27:51 2011
@@ -61,13 +61,13 @@
http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.9/apache-cassandra-0.7.9-bin.tar.gz";>apache-cassandra-0.7.9-bin.tar.gz
[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-bin.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-bin.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-bin.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-bin.tar.gz.sha1";>SHA1]
http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.9/apache-cassandra-0.7.9-src.tar.gz";>apache-cassandra-0.7.9-src.tar.gz
[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-src.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-src.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-src.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/0.7.9/apache-cassandra-0.7.9-src.tar.gz.sha1";>SHA1]
@@ -87,7 +87,7 @@
[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-bin.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-bin.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-bin.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-bin.tar.gz.sha1";>SHA1]
[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-src.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-src.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-src.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/0.8.6/apache-cassandra-0.8.6-src.tar.gz.sha1";>SHA1]
@@ -112,13 +112,13 @@
http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-bin.tar.gz";>apache-cassandra-1.0.0-beta1-bin.tar.gz
[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-bin.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-bin.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-bin.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-bin.tar.gz.sha1";>SHA1]
http://www.apache.org/dyn/closer.cgi?path=/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-src.tar.gz";>apache-cassandra-1.0.0-beta1-src.tar.gz
[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-src.tar.gz.asc";>PGP]
[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-src.tar.gz.md5";>MD5]
-[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-src.tar.gz.sha";>SHA1]
+[http://www.apache.org/dist/cassandra/1.0.0/apache-cassandra-1.0.0-beta1-src.tar.gz.sha1";>SHA1]
Modified: cassandra/site/src/content/download/index.html
URL:
http://svn.apache.org/viewvc/cassandra/site/src/content/download/index.html?rev=1174871&r1=1174870&r2=1174871&view=diff
==
--- cassandra/site/src/content/download/index.html (original)
+++ cassandra/site/src/content/download/index.html Fri Sep 23 16:27:51 2011
@@ -20,13 +20,13 @@
{{ oldbin_filename }}
[PGP]
[MD5]
-[SHA1]
+[SHA1]
{{ oldsrc_filename }}
[PGP]
[MD5]
-[SHA1]
+[SHA1]
{% endif %}
@@ -46,7 +46,7 @@
[PGP]
[MD5]
-[SHA1]
+[SHA1]
[PGP]
[MD5]
-[SHA1]
+[SHA1]
@@ -71,13 +71,13 @@
{{ devbin_filename }}
[PGP]
[MD5]
-[SHA1]
+[SHA1]
{{ devsrc_filename }}
[PGP]
[MD5]
-[SHA1]
+[SHA1]
{% endif %}
[jira] [Created] (CASSANDRA-3251) CassandraStorage uses comparator for both super column names and sub column names.
CassandraStorage uses comparator for both super column names and sub column names. -- Key: CASSANDRA-3251 URL: https://issues.apache.org/jira/browse/CASSANDRA-3251 Project: Cassandra Issue Type: Bug Components: Contrib, Hadoop Affects Versions: 0.8.6 Reporter: Dana H. P'Simer, Jr. The CassandraStorage class uses the same comparator for super and sub column names. This is because it calls columnsToTuple recursively without any indication that the subsequent call is for sub columns. Also, the getDefaultMarshallers method does not return the sub column name comparator. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113501#comment-13113501 ] chris erway edited comment on CASSANDRA-2863 at 9/23/11 3:27 PM: - Since my last two comments, I increased my RF and started a rolling repair on my nodes. This has caused this NPE to pop up on all the boxes over the last couple of days as they process SSTables. Again, all the nodes are fresh 0.8.6 installs from a few days ago using the ComboAMI. I've seen backtraces like the one below appear at least a couple of times on each node in my cluster as I was repairing.. INFO [CompactionExecutor:648] 2011-09-22 04:35:51,086 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF1-g-535 INFO [CompactionExecutor:648] 2011-09-22 04:35:51,172 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF2-g-350 INFO [CompactionExecutor:648] 2011-09-22 04:36:01,721 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF3-g-456 ERROR [Thread-3658] 2011-09-22 04:36:04,821 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-3658,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138) ... 3 more Caused by: java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [CompactionExecutor:648] 2011-09-22 04:36:04,823 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:648,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) was (Author: cce): Since my last two comments, I increased my RF and started a rolling repair on my nodes. This has caused this NPE to pop up on all the boxes over the last couple of days as they process SSTables. Again, all the nodes are fresh 0.8.6 installs from a few days ago using the ComboAMI. INFO [CompactionExecutor:648] 2011-09-22 04:35:51,086 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF1-g-535 INFO [CompactionExecutor:648] 2011-09-22 04:35:51,172 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF2-g-350 INFO [CompactionExecutor:648] 2011-09-22 04:36:01,721 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF3-g-456 ERROR [Thread-3658] 2011-09-22 04:36:04,821 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-3658,5,main] java.lang.RuntimeException: java.uti
[jira] [Commented] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113503#comment-13113503 ] chris erway commented on CASSANDRA-2863: Here's another from a node that was streaming out: INFO [AntiEntropyStage:1] 2011-09-22 17:56:33,737 StreamOut.java (line 181) Stream context metadata [/raid0/cassandra/data/Keyspace/CF1-g-315-Data.db sections=1073 progress=0/358 INFO [AntiEntropyStage:1] 2011-09-22 17:56:33,738 StreamOutSession.java (line 174) Streaming to /10.207.38.196 INFO [ValidationExecutor:14] 2011-09-22 17:56:33,740 ColumnFamilyStore.java (line 1128) Enqueuing flush of Memtable-CF4@516894197(9505947/21294113 serialized/live bytes, 5684 ops) INFO [FlushWriter:621] 2011-09-22 17:56:33,743 Memtable.java (line 237) Writing Memtable-CF4@516894197(9505947/21294113 serialized/live bytes, 5684 ops) INFO [FlushWriter:621] 2011-09-22 17:56:33,880 Memtable.java (line 254) Completed flushing /raid0/cassandra/data/Keyspace/CF4-g-434-Data.db (9914202 bytes) INFO [CompactionExecutor:884] 2011-09-22 17:56:45,912 CompactionManager.java (line 608) Compacted to /raid0/cassandra/data/Keyspace/CF4-tmp-g-332-Data.db. 32,824,572,147 to 18,506,0 ERROR [CompactionExecutor:884] 2011-09-22 17:56:46,134 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:884,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) > NPE when writing SSTable generated via repair > - > > Key: CASSANDRA-2863 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2863 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.1 >Reporter: Héctor Izquierdo > > A NPE is generated during repair when closing an sstable generated via > SSTable build. It doesn't happen always. The node had been scrubbed and > compacted before calling repair. > INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line > 158) Opening /d2/cassandra/data/sbs/walf-g-730 > ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:2,1,main] > java.lang.NullPointerException > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1103) > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1094) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113501#comment-13113501 ] chris erway edited comment on CASSANDRA-2863 at 9/23/11 3:18 PM: - Since my last two comments, I increased my RF and started a rolling repair on my nodes. This has caused this NPE to pop up on all the boxes over the last couple of days as they process SSTables. Again, all the nodes are fresh 0.8.6 installs from a few days ago using the ComboAMI. INFO [CompactionExecutor:648] 2011-09-22 04:35:51,086 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF1-g-535 INFO [CompactionExecutor:648] 2011-09-22 04:35:51,172 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF2-g-350 INFO [CompactionExecutor:648] 2011-09-22 04:36:01,721 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF3-g-456 ERROR [Thread-3658] 2011-09-22 04:36:04,821 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-3658,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138) ... 3 more Caused by: java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [CompactionExecutor:648] 2011-09-22 04:36:04,823 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:648,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) was (Author: cce): Since my last two comments, I increased my RF and started a rolling repair on my nodes. This has caused this NPE to pop up on all the boxes as the process SSTables. Again, all the nodes are fresh 0.8.6 installs from a few days ago using the ComboAMI. INFO [CompactionExecutor:648] 2011-09-22 04:35:51,086 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF1-g-535 INFO [CompactionExecutor:648] 2011-09-22 04:35:51,172 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF2-g-350 INFO [CompactionExecutor:648] 2011-09-22 04:36:01,721 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF3-g-456 ERROR [Thread-3658] 2011-09-22 04:36:04,821 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-3658,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSessio
[jira] [Commented] (CASSANDRA-2863) NPE when writing SSTable generated via repair
[ https://issues.apache.org/jira/browse/CASSANDRA-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113501#comment-13113501 ] chris erway commented on CASSANDRA-2863: Since my last two comments, I increased my RF and started a rolling repair on my nodes. This has caused this NPE to pop up on all the boxes as the process SSTables. Again, all the nodes are fresh 0.8.6 installs from a few days ago using the ComboAMI. INFO [CompactionExecutor:648] 2011-09-22 04:35:51,086 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF1-g-535 INFO [CompactionExecutor:648] 2011-09-22 04:35:51,172 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF2-g-350 INFO [CompactionExecutor:648] 2011-09-22 04:36:01,721 SSTableReader.java (line 162) Opening /raid0/cassandra/data/Keyspace/CF3-g-456 ERROR [Thread-3658] 2011-09-22 04:36:04,821 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-3658,5,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:154) at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63) at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:189) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:138) ... 3 more Caused by: java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [CompactionExecutor:648] 2011-09-22 04:36:04,823 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:648,1,main] java.lang.NullPointerException at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) at org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:315) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1108) at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:1099) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) > NPE when writing SSTable generated via repair > - > > Key: CASSANDRA-2863 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2863 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.1 >Reporter: Héctor Izquierdo > > A NPE is generated during repair when closing an sstable generated via > SSTable build. It doesn't happen always. The node had been scrubbed and > compacted before calling repair. > INFO [CompactionExecutor:2] 2011-07-06 11:11:32,640 SSTableReader.java (line > 158) Opening /d2/cassandra/data/sbs/walf-g-730 > ERROR [CompactionExecutor:2] 2011-07-06 11:11:34,327 > AbstractCassandraDaemon.java (line 113) Fatal exception in thread > Thread[CompactionExecutor:2,1,main] > java.lang.NullPointerException > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.close(SSTableWriter.java:382) > at > org.apache.cassandra.io.sstable.SSTableWriter$RowIndexer.index(SSTableWriter.java:370) > at > o
[jira] [Commented] (CASSANDRA-3214) Make CFIF use rpc_endpoint prior to trying endpoint
[ https://issues.apache.org/jira/browse/CASSANDRA-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113493#comment-13113493 ] Brandon Williams commented on CASSANDRA-3214: - It's (possibly) useful in the case that rpc_address is bound to all interfaces, because 0.0.0.0 is not useable. > Make CFIF use rpc_endpoint prior to trying endpoint > --- > > Key: CASSANDRA-3214 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3214 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Affects Versions: 0.8.6 > Environment: Hadoop 0.20 / Cassandra 0.8.5 / Ubuntu 10.04 / >Reporter: Eldon Stegall >Assignee: Eldon Stegall >Priority: Minor > Fix For: 0.8.7, 1.0.0 > > Attachments: CASSANDRA-3214-make-cfif-use-rpc-endpoints-v1.txt > > > Following up on CASSANDRA-3187 , we probably need to attempt to use the > rpc_endpoint address first, and then fall back to the gossip endpoint if we > don't get what we want. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3250) fsync the directory after new sstable or commit log segment are created
[ https://issues.apache.org/jira/browse/CASSANDRA-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113488#comment-13113488 ] Zhu Han commented on CASSANDRA-3250: There is a relative discussion on PostgreSQL. http://postgresql.1045698.n5.nabble.com/fsync-reliability-td4330289.html > fsync the directory after new sstable or commit log segment are created > --- > > Key: CASSANDRA-3250 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3250 > Project: Cassandra > Issue Type: Bug >Reporter: Zhu Han > > The mannual of fsync said: > bq. Calling fsync() does not necessarily ensure that the entry in > the directory containing the file has also reached disk. For that an > explicit fsync() on a file descriptor for the directory is also needed. > At least on ext4, syncing the directory is a must to have step, as described > by [1]. Otherwise, the new sstables or commit logs could be missed after > crash even if itself is synced. > Unfortunately, JVM does not provide an approach to sync the directory... > [1] > http://www.linuxfoundation.org/news-media/blogs/browse/2009/03/don%E2%80%99t-fear-fsync -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3250) fsync the directory after new sstable or commit log segment are created
fsync the directory after new sstable or commit log segment are created --- Key: CASSANDRA-3250 URL: https://issues.apache.org/jira/browse/CASSANDRA-3250 Project: Cassandra Issue Type: Bug Reporter: Zhu Han The mannual of fsync said: bq. Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed. At least on ext4, syncing the directory is a must to have step, as described by [1]. Otherwise, the new sstables or commit logs could be missed after crash even if itself is synced. Unfortunately, JVM does not provide an approach to sync the directory... [1] http://www.linuxfoundation.org/news-media/blogs/browse/2009/03/don%E2%80%99t-fear-fsync -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113466#comment-13113466
]
Zhu Han commented on CASSANDRA-3248:
It is very strange that there is no different between fdatasync and fsync when
overwrite an preallocated file. It should be highly correlated with the
underlying file system implementation.
IMHO, we should not fix this issue unless we have a thorough test on different
file systems.
{quote}
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=512M
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqrewr
--max-time=600 --file-block-size=2K --max-requests=0 prepare
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=512M
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqrewr --max-time=600 --file-block-size=2K --max-requests=0
run
Operations performed: 0 Read, 29384 Write, 29384 Other = 58768 Total
Read 0b Written 57.391Mb Total transferred 57.391Mb (97.943Kb/sec)
48.97 Requests/sec executed
per-request statistics:
min: 12.94ms
avg: 20.42ms
max:125.02ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=512M
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqrewr --max-time=600 --file-block-size=2K --max-requests=0
run
Operations performed: 0 Read, 29307 Write, 29307 Other = 58614 Total
Read 0b Written 57.24Mb Total transferred 57.24Mb (97.688Kb/sec)
48.84 Requests/sec executed
per-request statistics:
min: 16.21ms
avg: 20.47ms
max:116.69ms
approx. 95 percentile: 25.02ms
{quote}
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical
[jira] [Issue Comment Edited] (CASSANDRA-3244) JDBC CassandraConnection may lead to memory leak when used in a pool
[
https://issues.apache.org/jira/browse/CASSANDRA-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113452#comment-13113452
]
Rick Shaw edited comment on CASSANDRA-3244 at 9/23/11 2:11 PM:
---
Creating a {{Statement}} every time you need one is unnecessary but not
illegal. The assumption is the {{Statement}} is closed when Spring is done,
which should be just fine. The problem is the list in {{Connection}} _should_
be a list of open {{Statements}}. The implication being that closing a
{{Statement}} removes itself from that list. Unfortunately this is not being
done.
The list is necessary. But it needs to be maintained by {{Statement}} because
only it knows when it is asked to close.
Thanks for catching this bug. I'll create a fix for this.
was (Author: ardot):
Creating a {{Statement}} every time you need one is unnecessary but not
illegal. The assumption is the {{Statement}} is closed when Spring is done,
which should be just fine. The problem is the list in {{Connection}} _should_
be a list of open {{Statements}}. The implication being that closing a
{{Statement}} removes itself from that list. Unfortunately this is not being
done.
The list is necessary. But it needs to be maintained by {{Statement}} because
only it knows when it is asked to close.
As a note this has nothing to do with pooling. The JDBC driver pooling
({{PooledConnection}}) will handle the issues around {{Statement}} creation and
does not engage the physical {{Connection}} so the connection can be reused
because none of that state is placed at the physical level and is therefore
reusable without closing it.
Thanks for catching this bug. I'll create a fix for this.
> JDBC CassandraConnection may lead to memory leak when used in a pool
>
>
> Key: CASSANDRA-3244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3244
> Project: Cassandra
> Issue Type: Improvement
> Components: Drivers
>Affects Versions: 1.0.0
>Reporter: Patricio Echague
>Priority: Minor
>
> I may be wrong here but I noticed that the implementations of
> CassandraConnection#createStatement() and
> CassandraConnection#prepareStatement() keep(cache) the created
> Statement/PrepareStatement internally in a List.
> They list is freed up only during CassandraConnection.close() which makes me
> think that, if the connection object is used in a pool implementation, it
> will lead to a memory leak as it will hold every single statement that is
> used to interact with the DB until the connection gets closed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3244) JDBC CassandraConnection may lead to memory leak when used in a pool
[
https://issues.apache.org/jira/browse/CASSANDRA-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113452#comment-13113452
]
Rick Shaw edited comment on CASSANDRA-3244 at 9/23/11 2:04 PM:
---
Creating a {{Statement}} every time you need one is unnecessary but not
illegal. The assumption is the {{Statement}} is closed when Spring is done,
which should be just fine. The problem is the list in {{Connection}} _should_
be a list of open {{Statements}}. The implication being that closing a
{{Statement}} removes itself from that list. Unfortunately this is not being
done.
The list is necessary. But it needs to be maintained by {{Statement}} because
only it knows when it is asked to close.
As a note this has nothing to do with pooling. The JDBC driver pooling
({{PooledConnection}}) will handle the issues around {{Statement}} creation and
does not engage the physical {{Connection}} so the connection can be reused
because none of that state is placed at the physical level and is therefore
reusable without closing it.
Thanks for catching this bug. I'll create a fix for this.
was (Author: ardot):
Creating a {{Statement}} every time you need one is unnecessary but not
illegal. The assumption is the {{Statement}} is closed when Spring is done,
which should be just fine. The problem is the list in {{Connection}} _should_
be a list of open {{Statements}}. The implecation being that closing a
{{Statement}} removes itself from that list. Unfortunately this is not being
done.
The list is necessary. But it needs to be maintained by {{Statement}} because
only it knows when it is asked to close.
As a note this has nothing to do with pooling. The JDBC driver pooling
({{PooledConnection}}) will handle the issues around {{Statement}} creation and
does not engage the physical {{Connection}} so the connection can be reused
because none of that state is placed at the physical level and is therefore
reusable without closing it.
Thanks for catching this bug. I'll create a fix for this.
> JDBC CassandraConnection may lead to memory leak when used in a pool
>
>
> Key: CASSANDRA-3244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3244
> Project: Cassandra
> Issue Type: Improvement
> Components: Drivers
>Affects Versions: 1.0.0
>Reporter: Patricio Echague
>Priority: Minor
>
> I may be wrong here but I noticed that the implementations of
> CassandraConnection#createStatement() and
> CassandraConnection#prepareStatement() keep(cache) the created
> Statement/PrepareStatement internally in a List.
> They list is freed up only during CassandraConnection.close() which makes me
> think that, if the connection object is used in a pool implementation, it
> will lead to a memory leak as it will hold every single statement that is
> used to interact with the DB until the connection gets closed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3244) JDBC CassandraConnection may lead to memory leak when used in a pool
[
https://issues.apache.org/jira/browse/CASSANDRA-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113452#comment-13113452
]
Rick Shaw commented on CASSANDRA-3244:
--
Creating a {{Statement}} every time you need one is unnecessary but not
illegal. The assumption is the {{Statement}} is closed when Spring is done,
which should be just fine. The problem is the list in {{Connection}} _should_
be a list of open {{Statements}}. The implecation being that closing a
{{Statement}} removes itself from that list. Unfortunately this is not being
done.
The list is necessary. But it needs to be maintained by {{Statement}} because
only it knows when it is asked to close.
As a note this has nothing to do with pooling. The JDBC driver pooling
({{PooledConnection}}) will handle the issues around {{Statement}} creation and
does not engage the physical {{Connection}} so the connection can be reused
because none of that state is placed at the physical level and is therefore
reusable without closing it.
Thanks for catching this bug. I'll create a fix for this.
> JDBC CassandraConnection may lead to memory leak when used in a pool
>
>
> Key: CASSANDRA-3244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3244
> Project: Cassandra
> Issue Type: Improvement
> Components: Drivers
>Affects Versions: 1.0.0
>Reporter: Patricio Echague
>Priority: Minor
>
> I may be wrong here but I noticed that the implementations of
> CassandraConnection#createStatement() and
> CassandraConnection#prepareStatement() keep(cache) the created
> Statement/PrepareStatement internally in a List.
> They list is freed up only during CassandraConnection.close() which makes me
> think that, if the connection object is used in a pool implementation, it
> will lead to a memory leak as it will hold every single statement that is
> used to interact with the DB until the connection gets closed.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1174770 - in /cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db: ./ compaction/
Author: jbellis
Date: Fri Sep 23 13:58:54 2011
New Revision: 1174770
URL: http://svn.apache.org/viewvc?rev=1174770&view=rev
Log:
avoid echoedRow when checking shouldPurge is more expensive than just
de/serializing
patch by jbellis; reviewed by slebresne for CASSANDRA-3234
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionController.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=1174770&r1=1174769&r2=1174770&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
Fri Sep 23 13:58:54 2011
@@ -885,6 +885,11 @@ public class ColumnFamilyStore implement
return false;
}
+public boolean isKeyExistenceExpensive(Set
sstablesToIgnore)
+{
+return compactionStrategy.isKeyExistenceExpensive(sstablesToIgnore);
+}
+
/*
* Called after a BinaryMemtable flushes its in-memory data, or we add a
file
* via bootstrap. This information is cached in the ColumnFamilyStore.
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java?rev=1174770&r1=1174769&r2=1174770&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
Fri Sep 23 13:58:54 2011
@@ -21,9 +21,11 @@ package org.apache.cassandra.db.compacti
import java.util.Collection;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.io.sstable.SSTable;
import org.apache.cassandra.io.sstable.SSTableReader;
import org.apache.cassandra.service.StorageService;
@@ -97,4 +99,10 @@ public abstract class AbstractCompaction
* @return size in bytes of the largest sstables for this strategy
*/
public abstract long getMaxSSTableSize();
+
+/**
+ * @return true if checking for whether a key exists, ignoring @param
sstablesToIgnore,
+ * is going to be expensive
+ */
+public abstract boolean isKeyExistenceExpensive(Set
sstablesToIgnore);
}
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionController.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionController.java?rev=1174770&r1=1174769&r2=1174770&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionController.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionController.java
Fri Sep 23 13:58:54 2011
@@ -44,6 +44,7 @@ public class CompactionController
private final boolean forceDeserialize;
public final int gcBefore;
+public boolean keyExistenceIsExpensive;
public CompactionController(ColumnFamilyStore cfs,
Collection sstables, int gcBefore, boolean forceDeserialize)
{
@@ -52,6 +53,7 @@ public class CompactionController
this.sstables = new HashSet(sstables);
this.gcBefore = gcBefore;
this.forceDeserialize = forceDeserialize;
+keyExistenceIsExpensive =
cfs.getCompactionStrategy().isKeyExistenceExpensive(this.sstables);
}
public String getKeyspace()
@@ -102,13 +104,19 @@ public class CompactionController
*/
public AbstractCompactedRow getCompactedRow(List
rows)
{
-if (rows.size() == 1 && !needDeserialize() &&
!shouldPurge(rows.get(0).getKey()))
-return new EchoedRow(this, rows.get(0));
-
svn commit: r1174766 - in /cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db: ColumnFamilyStore.java compaction/CompactionTask.java compaction/LeveledCompactionStrategy.java compacti
Author: jbellis
Date: Fri Sep 23 13:55:55 2011
New Revision: 1174766
URL: http://svn.apache.org/viewvc?rev=1174766&view=rev
Log:
cleanup LCS logging
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=1174766&r1=1174765&r2=1174766&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
Fri Sep 23 13:55:55 2011
@@ -321,7 +321,8 @@ public class ColumnFamilyStore implement
*/
public static void scrubDataDirectories(String table, String columnFamily)
{
-logger.info("Removing compacted SSTable files from " + columnFamily +
" (see http://wiki.apache.org/cassandra/MemtableSSTable)");
+logger.debug("Removing compacted SSTable files from {} (see
http://wiki.apache.org/cassandra/MemtableSSTable)", columnFamily);
+
for (Map.Entry> sstableFiles : files(table,
columnFamily, true, true).entrySet())
{
Descriptor desc = sstableFiles.getKey();
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java?rev=1174766&r1=1174765&r2=1174766&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
Fri Sep 23 13:55:55 2011
@@ -72,11 +72,12 @@ public class CompactionTask extends Abst
Set toCompact = new HashSet(sstables);
if (!isUserDefined)
{
-if ( !allowSingletonCompaction() && toCompact.size() < 2)
+if (!allowSingletonCompaction() && toCompact.size() < 2)
{
-logger.info("Nothing to compact in " +
cfs.getColumnFamilyName() + "." +
-"Use forceUserDefinedCompaction if you wish to
force compaction of single sstables " +
-"(e.g. for tombstone collection)");
+String msg = "Nothing to compact in " +
cfs.getColumnFamilyName();
+if (cfs.getCompactionStrategy() instanceof
SizeTieredCompactionStrategy)
+msg += ". Use forceUserDefinedCompaction if you wish to
force compaction of single sstables (e.g. for tombstone collection)";
+logger.info(msg);
return 0;
}
@@ -220,7 +221,7 @@ public class CompactionTask extends Abst
double mbps = dTime > 0 ?
(double)endsize/(1024*1024)/((double)dTime/1000) : 0;
logger.info(String.format("Compacted to %s. %,d to %,d (~%d%% of
original) bytes for %,d keys at %fMBPS. Time: %,dms.",
builder.toString(), startsize, endsize,
(int) (ratio * 100), totalkeysWritten, mbps, dTime));
-logger.info(String.format("CF Total Bytes Compacted: %,d",
CompactionTask.addToTotalBytesCompacted(endsize)));
+logger.debug(String.format("CF Total Bytes Compacted: %,d",
CompactionTask.addToTotalBytesCompacted(endsize)));
return toCompact.size();
}
Modified:
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java?rev=1174766&r1=1174765&r2=1174766&view=diff
==
---
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
(original)
+++
cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
Fri Sep 23 13:55:55 2011
@@ -71,7 +71,7 @@ public class LeveledCompactionStrategy e
maxSSTableSizeInMB = configuredMaxSSTableSize;
cfs.getDataTracker().subscribe(this);
-logger.info(this + " subscribed to the data tracker.");
+logger.debug("{} subscribed to the data tracker.", t
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113437#comment-13113437
]
Zhu Han commented on CASSANDRA-3248:
I checked the source code of XFS which is used on my log device. A recent patch
optimizes the fdatasync() path[1].
I am sure fdatasync() flushes the inode (with length information) to disk if
the EOF of file is extended, at least on XFS[2].
>From xfs code, fsync() and fdatasync() should not have much difference for
>append-only write. But I do observe the difference here...
I will do another test to see whether there is different between fsync and
fdatasync when overwrite an existing file.
[1]http://oss.sgi.com/archives/xfs/2010-02/msg00236.html
[2]http://lxr.linux.no/linux+*/fs/xfs/linux-2.6/xfs_aops.c#L400
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3249) Index search in provided set of rows (support of sub query)
[
https://issues.apache.org/jira/browse/CASSANDRA-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Ryabitskiy updated CASSANDRA-3249:
-
Summary: Index search in provided set of rows (support of sub query) (was:
Index search in provided set of rows (supporting of sub query))
> Index search in provided set of rows (support of sub query)
> ---
>
> Key: CASSANDRA-3249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3249
> Project: Cassandra
> Issue Type: New Feature
> Components: API, Core
>Affects Versions: 0.8.6
>Reporter: Evgeny Ryabitskiy
> Fix For: 1.1
>
>
> This issue is related to discussion on mailing list:
> http://www.mail-archive.com/user@cassandra.apache.org/msg17135.html
> Idea is to support Cassandra build-in index search over specified set of rows.
> From API view:
> It can be extension of get_indexed_slices, for example:
> {quote}
> List rowKys = ... ; //list of row keys
> IndexClause indexClause = new IndexClause();
> indexClause.setKeys(keys); //required API to set list of keys
> indexClause.setExpressions(someFilteringExpressions);
> List finalResult = get_indexed_slices(colParent, indexClause, colPredicate,
> cLevel);
> {quote}
> or create specified API method.
> From conceptual view it was noticed:
> That would be implementation of sub query.
> {quote}
> The index clause is applied to the set of all rows in the database, not a sub
> set, applying them to a sub set is implicitly supporting a sub query
> {quote}
> Benefits of this feature is that search can be split in 2 stages:
> 1) Search over external engine (for example full text search)
> 2) Cassandra build-in index search over result from first stage
> This combination could solve most of limitations that came with solution
> based only on external search engine.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3249) Index search in provided set of rows (supporting of sub query)
Index search in provided set of rows (supporting of sub query)
--
Key: CASSANDRA-3249
URL: https://issues.apache.org/jira/browse/CASSANDRA-3249
Project: Cassandra
Issue Type: New Feature
Components: API, Core
Affects Versions: 0.8.6
Reporter: Evgeny Ryabitskiy
Fix For: 1.1
This issue is related to discussion on mailing list:
http://www.mail-archive.com/user@cassandra.apache.org/msg17135.html
Idea is to support Cassandra build-in index search over specified set of rows.
>From API view:
It can be extension of get_indexed_slices, for example:
{quote}
List rowKys = ... ; //list of row keys
IndexClause indexClause = new IndexClause();
indexClause.setKeys(keys); //required API to set list of keys
indexClause.setExpressions(someFilteringExpressions);
List finalResult = get_indexed_slices(colParent, indexClause, colPredicate,
cLevel);
{quote}
or create specified API method.
>From conceptual view it was noticed:
That would be implementation of sub query.
{quote}
The index clause is applied to the set of all rows in the database, not a sub
set, applying them to a sub set is implicitly supporting a sub query
{quote}
Benefits of this feature is that search can be split in 2 stages:
1) Search over external engine (for example full text search)
2) Cassandra build-in index search over result from first stage
This combination could solve most of limitations that came with solution based
only on external search engine.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3247) sstableloader ignores option doesn't work correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113426#comment-13113426 ] Hudson commented on CASSANDRA-3247: --- Integrated in Cassandra-0.8 #338 (See [https://builds.apache.org/job/Cassandra-0.8/338/]) fix sstableloader --ignores option patch by slebresne; reviewed by jbellis for CASSANDRA-3247 slebresne : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1174701 Files : * /cassandra/branches/cassandra-0.8/CHANGES.txt * /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java > sstableloader ignores option doesn't work correctly > --- > > Key: CASSANDRA-3247 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3247 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: bulkloader > Fix For: 0.8.7, 1.0.0 > > Attachments: 3247.patch > > > The --ignores option is supposed to take an argument but it doesn't. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113397#comment-13113397
]
Jonathan Ellis commented on CASSANDRA-3248:
---
Postgresql can do that because they pre-allocate fixed-size CL segments. See
CASSANDRA-622.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113388#comment-13113388
]
Zhu Han edited comment on CASSANDRA-3248 at 9/23/11 12:56 PM:
--
The manual said:
{quote}
$man fdatasync
fdatasync() is similar to fsync(), but does not flush modified metadata unless
that metadata is needed in order to allow a subsequent data retrieval to be
correctly handled.
{quote}
File size should be the metadata which is needed to read the data correctly.
was (Author: hanzhu):
The manual said:
{quote}
$man fdatasync
fdatasync() is similar to fsync(), but does not flush modified metadata unless
that metadata is needed in order to allow a subsequent data retrieval to be
correctly handled.
On the other hand, a change to the file size (st_size, as made by say
ftruncate(2)), would require a metadata flush
{quote}
File size should be the metadata which is needed to read the data correctly.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113392#comment-13113392
]
Zhu Han commented on CASSANDRA-3248:
And postgresql uses fdatasync() to flush the write ahead log[1]. I will take a
look at the kernel implementation.
[1]http://www.filehippo.com/es/download_postgresql/changelog/8968/
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113388#comment-13113388
]
Zhu Han commented on CASSANDRA-3248:
The manual said:
{quote}
$man fdatasync
fdatasync() is similar to fsync(), but does not flush modified metadata unless
that metadata is needed in order to allow a subsequent data retrieval to be
correctly handled.
On the other hand, a change to the file size (st_size, as made by say
ftruncate(2)), would require a metadata flush
{quote}
File size should be the metadata which is needed to read the data correctly.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment uses SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Description:
CommitLogSegment uses SequentialWriter to flush the buffered data to log
device. It depends on FileDescriptor#sync() which invokes fsync() as it force
the file attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}54.90{color} Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}28.08{color} Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
was:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}54.90{color} Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}28.08{color} Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
> Com
[jira] [Commented] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113380#comment-13113380
]
Jonathan Ellis commented on CASSANDRA-3248:
---
"File size is synced to disk by fdatasync"
Are you sure?
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment use SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Affects Version/s: 0.6.13
0.7.9
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3248
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.6.13, 0.7.9, 0.8.6, 1.0.0, 1.1
> Environment: Linux
>Reporter: Zhu Han
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> CommitLogSegment use SequentialWriter to flush the buffered data to log
> device. It depends on FileDescriptor#sync() which invokes fsync() as it force
> the file attributes to disk.
> However, at least on Linux, fdatasync() is good enough for commit log flush:
> bq. fdatasync() is similar to fsync(), but does not flush modified metadata
> unless that metadata is needed in order to allow a subsequent data retrieval
> to be correctly handled. For example, changes to st_atime or st_mtime
> (respectively, time of last access and time of last modification; see
> stat(2)) do not require flushing because they are not necessary for a
> subsequent data read to be handled correctly. On the other hand, a change to
> the file size (st_size, as made by say ftruncate(2)), would require a
> metadata flush.
> File size is synced to disk by fdatasync() either. Although the commit log
> recovery logic sorts the commit log segements on their modify timestamp, it
> can be removed safely, IMHO.
> I checked the native code of JRE 6. On Linux and Solaris,
> FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
> not have any impact.
> On my log device (commodity SATA HDD, write cache disabled), there is large
> performance gap between fsync() and fdatasync():
> {quote}
> $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}54.90{color} Requests/sec executed
>per-request statistics:
> min: 8.29ms
> avg: 18.18ms
> max:108.36ms
> approx. 95 percentile: 25.02ms
> $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
> --file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
> --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0
> run
> {color:blue}28.08{color} Requests/sec executed
> per-request statistics:
> min: 33.28ms
> avg: 35.61ms
> max:911.87ms
> approx. 95 percentile: 41.69ms
> {quote}
> I do think this is a very critical performance improvement.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Description:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}54.90{color} Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
{color:blue}28.08{color} Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
was:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
> CommitLog writer should call fdatasync inst
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Description:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fdatasync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode={color:red}fsync{color}
--file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
was:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
>
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Description:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), there is large
performance gap between fsync() and fdatasync():
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
was:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), fsync() and
fdatasync() has large performance gap:
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
>
[jira] [Updated] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
[
https://issues.apache.org/jira/browse/CASSANDRA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Han updated CASSANDRA-3248:
---
Description:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), fsync() and
fdatasync() has large performance gap:
{quote}
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
{quote}
I do think this is a very critical performance improvement.
was:
CommitLogSegment use SequentialWriter to flush the buffered data to log device.
It depends on FileDescriptor#sync() which invokes fsync() as it force the file
attributes to disk.
However, at least on Linux, fdatasync() is good enough for commit log flush:
bq. fdatasync() is similar to fsync(), but does not flush modified metadata
unless that metadata is needed in order to allow a subsequent data retrieval to
be correctly handled. For example, changes to st_atime or st_mtime
(respectively, time of last access and time of last modification; see stat(2))
do not require flushing because they are not necessary for a subsequent data
read to be handled correctly. On the other hand, a change to the file size
(st_size, as made by say ftruncate(2)), would require a metadata flush.
File size is synced to disk by fdatasync() either. Although the commit log
recovery logic sorts the commit log segements on their modify timestamp, it can
be removed safely, IMHO.
I checked the native code of JRE 6. On Linux and Solaris,
FileChannel#force(false) invokes fdatasync(). On windows, the false flag does
not have any impact.
On my log device (commodity SATA HDD, write cache disabled), fsync() and
fdatasync() has large performance gap:
$sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
54.90 Requests/sec executed
per-request statistics:
min: 8.29ms
avg: 18.18ms
max:108.36ms
approx. 95 percentile: 25.02ms
$ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G
--file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr
--max-time=600 --file-block-size=2K --max-requests=0 run
28.08 Requests/sec executed
per-request statistics:
min: 33.28ms
avg: 35.61ms
max:911.87ms
approx. 95 percentile: 41.69ms
I do think this is a very critical performance improvement.
> CommitLog writer should call fdatasync instead of fsync
> ---
>
> Key: CASSANDRA-3248
> URL: https://issues.apac
[jira] [Created] (CASSANDRA-3248) CommitLog writer should call fdatasync instead of fsync
CommitLog writer should call fdatasync instead of fsync --- Key: CASSANDRA-3248 URL: https://issues.apache.org/jira/browse/CASSANDRA-3248 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.6, 1.0.0, 1.1 Environment: Linux Reporter: Zhu Han CommitLogSegment use SequentialWriter to flush the buffered data to log device. It depends on FileDescriptor#sync() which invokes fsync() as it force the file attributes to disk. However, at least on Linux, fdatasync() is good enough for commit log flush: bq. fdatasync() is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled. For example, changes to st_atime or st_mtime (respectively, time of last access and time of last modification; see stat(2)) do not require flushing because they are not necessary for a subsequent data read to be handled correctly. On the other hand, a change to the file size (st_size, as made by say ftruncate(2)), would require a metadata flush. File size is synced to disk by fdatasync() either. Although the commit log recovery logic sorts the commit log segements on their modify timestamp, it can be removed safely, IMHO. I checked the native code of JRE 6. On Linux and Solaris, FileChannel#force(false) invokes fdatasync(). On windows, the false flag does not have any impact. On my log device (commodity SATA HDD, write cache disabled), fsync() and fdatasync() has large performance gap: $sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G --file-fsync-all=on --file-fsync-mode=fdatasync --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run 54.90 Requests/sec executed per-request statistics: min: 8.29ms avg: 18.18ms max:108.36ms approx. 95 percentile: 25.02ms $ sysbench --test=fileio --num-threads=1 --file-num=1 --file-total-size=10G --file-fsync-all=on --file-fsync-mode=fsync --file-test-mode=seqwr --max-time=600 --file-block-size=2K --max-requests=0 run 28.08 Requests/sec executed per-request statistics: min: 33.28ms avg: 35.61ms max:911.87ms approx. 95 percentile: 41.69ms I do think this is a very critical performance improvement. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1174708 - in /cassandra/trunk: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/tools/
Author: slebresne Date: Fri Sep 23 12:35:33 2011 New Revision: 1174708 URL: http://svn.apache.org/viewvc?rev=1174708&view=rev Log: merge from 1.0 Modified: cassandra/trunk/ (props changed) cassandra/trunk/CHANGES.txt cassandra/trunk/contrib/ (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/trunk/src/java/org/apache/cassandra/tools/BulkLoader.java Propchange: cassandra/trunk/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:35:33 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 -/cassandra/branches/cassandra-1.0:1167085-1174478 -/cassandra/branches/cassandra-1.0.0:1167104-1167229,1167232-1174472 +/cassandra/branches/cassandra-1.0:1167085-1174478,1174706 +/cassandra/branches/cassandra-1.0.0:1167104-1167229,1167232-1174472,1174704 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/tags/cassandra-0.8.0-rc1:1102511-1125020 /incubator/cassandra/branches/cassandra-0.3:774578-796573 Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1174708&r1=1174707&r2=1174708&view=diff == --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Fri Sep 23 12:35:33 2011 @@ -20,6 +20,7 @@ * fix handling of exceptions writing to OutboundTcpConnection (CASSANDRA-3235) * Allow using quotes in "USE ;" CLI command (CASSANDRA-3208) * Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218) + * Fix sstableloader --ignores option (CASSANDRA-3247) 1.0.0-beta1 Propchange: cassandra/trunk/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:35:33 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 -/cassandra/branches/cassandra-1.0/contrib:1167085-1174478 -/cassandra/branches/cassandra-1.0.0/contrib:1167104-1167229,1167232-1174472 +/cassandra/branches/cassandra-1.0/contrib:1167085-1174478,1174706 +/cassandra/branches/cassandra-1.0.0/contrib:1167104-1167229,1167232-1174472,1174704 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 /cassandra/tags/cassandra-0.8.0-rc1/contrib:1102511-1125020 /incubator/cassandra/branches/cassandra-0.3/contrib:774578-796573 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:35:33 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 -/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369 /cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.
svn commit: r1174706 - in /cassandra/branches/cassandra-1.0: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/tools/
Author: slebresne Date: Fri Sep 23 12:34:10 2011 New Revision: 1174706 URL: http://svn.apache.org/viewvc?rev=1174706&view=rev Log: merge from 1.0.0 Modified: cassandra/branches/cassandra-1.0/ (props changed) cassandra/branches/cassandra-1.0/CHANGES.txt cassandra/branches/cassandra-1.0/contrib/ (props changed) cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-1.0/src/java/org/apache/cassandra/tools/BulkLoader.java Propchange: cassandra/branches/cassandra-1.0/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:34:10 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 /cassandra/branches/cassandra-1.0:1167106,1167185 -/cassandra/branches/cassandra-1.0.0:1167104-1174472 +/cassandra/branches/cassandra-1.0.0:1167104-1174472,1174704 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 /cassandra/tags/cassandra-0.8.0-rc1:1102511-1125020 /cassandra/trunk:1167085-1167102,1169870 Modified: cassandra/branches/cassandra-1.0/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0/CHANGES.txt?rev=1174706&r1=1174705&r2=1174706&view=diff == --- cassandra/branches/cassandra-1.0/CHANGES.txt (original) +++ cassandra/branches/cassandra-1.0/CHANGES.txt Fri Sep 23 12:34:10 2011 @@ -20,6 +20,7 @@ * fix handling of exceptions writing to OutboundTcpConnection (CASSANDRA-3235) * Allow using quotes in "USE ;" CLI command (CASSANDRA-3208) * Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218) + * Fix sstableloader --ignores option (CASSANDRA-3247) 1.0.0-beta1 Propchange: cassandra/branches/cassandra-1.0/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:34:10 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 /cassandra/branches/cassandra-1.0/contrib:1167106,1167185 -/cassandra/branches/cassandra-1.0.0/contrib:1167104-1174472 +/cassandra/branches/cassandra-1.0.0/contrib:1167104-1174472,1174704 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 /cassandra/tags/cassandra-0.8.0-rc1/contrib:1102511-1125020 /cassandra/trunk/contrib:1167085-1167102,1169870 Propchange: cassandra/branches/cassandra-1.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:34:10 2011 @@ -1,11 +1,11 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 -/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369 /cassandra/branches/cassandra-0.8.1/interface/thr
svn commit: r1174704 - in /cassandra/branches/cassandra-1.0.0: ./ contrib/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/tools/
Author: slebresne Date: Fri Sep 23 12:32:28 2011 New Revision: 1174704 URL: http://svn.apache.org/viewvc?rev=1174704&view=rev Log: merge from 0.8 Modified: cassandra/branches/cassandra-1.0.0/ (props changed) cassandra/branches/cassandra-1.0.0/CHANGES.txt cassandra/branches/cassandra-1.0.0/contrib/ (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/branches/cassandra-1.0.0/src/java/org/apache/cassandra/tools/BulkLoader.java Propchange: cassandra/branches/cassandra-1.0.0/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:32:28 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0:1053690-1055654 -/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0:1125021-1130369 /cassandra/branches/cassandra-0.8.1:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689 Modified: cassandra/branches/cassandra-1.0.0/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/branches/cassandra-1.0.0/CHANGES.txt?rev=1174704&r1=1174703&r2=1174704&view=diff == --- cassandra/branches/cassandra-1.0.0/CHANGES.txt (original) +++ cassandra/branches/cassandra-1.0.0/CHANGES.txt Fri Sep 23 12:32:28 2011 @@ -16,6 +16,7 @@ * fix handling of exceptions writing to OutboundTcpConnection (CASSANDRA-3235) * Allow using quotes in "USE ;" CLI command (CASSANDRA-3208) * Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218) + * Fix sstableloader --ignores option (CASSANDRA-3247) 1.0.0-beta1 Propchange: cassandra/branches/cassandra-1.0.0/contrib/ -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:32:28 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009 /cassandra/branches/cassandra-0.7/contrib:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654 -/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/contrib:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/contrib:1125021-1130369 /cassandra/branches/cassandra-0.8.1/contrib:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689 Propchange: cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:32:28 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1131291 /cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1170333,1172024 /cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1053690-1055654 -/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469 +/cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1090934-1125013,1125019-1174469,1174701 /cassandra/branches/cassandra-0.8.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1125021-1130369 /cassandra/branches/cassandra-0.8.1/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1101014-1125018 /cassandra/tags/cassandra-0.7.0-rc3/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1051699-1053689 Propchange: cassandra/branches/cassandra-1.0.0/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java -- --- svn:mergeinfo (original) +++ svn:mergeinfo Fri Sep 23 12:32:28 2011 @@ -1,7 +1,7 @@ /cassandra/branches/cassandra-0.6/interface/thrift/gen-ja
svn commit: r1174701 - in /cassandra/branches/cassandra-0.8: CHANGES.txt src/java/org/apache/cassandra/tools/BulkLoader.java
Author: slebresne
Date: Fri Sep 23 12:31:02 2011
New Revision: 1174701
URL: http://svn.apache.org/viewvc?rev=1174701&view=rev
Log:
fix sstableloader --ignores option
patch by slebresne; reviewed by jbellis for CASSANDRA-3247
Modified:
cassandra/branches/cassandra-0.8/CHANGES.txt
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java
Modified: cassandra/branches/cassandra-0.8/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/CHANGES.txt?rev=1174701&r1=1174700&r2=1174701&view=diff
==
--- cassandra/branches/cassandra-0.8/CHANGES.txt (original)
+++ cassandra/branches/cassandra-0.8/CHANGES.txt Fri Sep 23 12:31:02 2011
@@ -3,6 +3,7 @@
* Allow using quotes in "USE ;" CLI command (CASSANDRA-3208)
* Log message when a full repair operation completes (CASSANDRA-3207)
* Don't allow any cache loading exceptions to halt startup (CASSANDRA-3218)
+ * Fix sstableloader --ignores option (CASSANDRA-3247)
0.8.6
Modified:
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java?rev=1174701&r1=1174700&r2=1174701&view=diff
==
---
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java
(original)
+++
cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/tools/BulkLoader.java
Fri Sep 23 12:31:02 2011
@@ -314,12 +314,12 @@ public class BulkLoader
{
for (String node : nodes)
{
-opts.ignores.add(InetAddress.getByName(node));
+
opts.ignores.add(InetAddress.getByName(node.trim()));
}
}
catch (UnknownHostException e)
{
-errorMsg(e.getMessage(), options);
+errorMsg("Unknown host: " + e.getMessage(), options);
}
}
@@ -357,7 +357,7 @@ public class BulkLoader
options.addOption("v", VERBOSE_OPTION, "verbose output");
options.addOption("h", HELP_OPTION, "display this help
message");
options.addOption(null, NOPROGRESS_OPTION, "don't display
progress");
-options.addOption("i", IGNORE_NODES_OPTION, "don't stream to this
(comma separated) list of nodes");
+options.addOption("i", IGNORE_NODES_OPTION, "NODES", "don't
stream to this (comma separated) list of nodes");
return options;
}
[jira] [Commented] (CASSANDRA-3247) sstableloader ignores option doesn't work correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113361#comment-13113361 ] Jonathan Ellis commented on CASSANDRA-3247: --- +1 (nit: "Unknow" should be "Unknown") > sstableloader ignores option doesn't work correctly > --- > > Key: CASSANDRA-3247 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3247 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: bulkloader > Fix For: 0.8.7, 1.0.0 > > Attachments: 3247.patch > > > The --ignores option is supposed to take an argument but it doesn't. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326
]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 12:05 PM:
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input
records when cassandra.input.split.size is set to 393216).
bq. ... double-check how many rows there really are in the given split range.
I can confirm that if i let these jobs run through to their completion (despite
how terribly long they may take) that the results are correct. It would seem
that the split ranges are incorrect (not the rows within in them).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the
day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input
records when cassandra.input.split.size is set to 393216).
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for
> task_201109212019_1060_m_29 - Mozilla Firefox.png, Screenshot-Hadoop map
> task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mck SembWever updated CASSANDRA-3150:
-
Attachment: Screenshot-Counters for task_201109212019_1060_m_29 -
Mozilla Firefox.png
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for
> task_201109212019_1060_m_29 - Mozilla Firefox.png, Screenshot-Hadoop map
> task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326
]
Mck SembWever edited comment on CASSANDRA-3150 at 9/23/11 11:04 AM:
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages (you see 36million map input
records when cassandra.input.split.size is set to 393216).
was (Author: michaelsembwever):
This is persisting to be a problem. And continuously gets worse through the
day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Counters for
> task_201109212019_1060_m_29 - Mozilla Firefox.png, Screenshot-Hadoop map
> task list for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mck SembWever updated CASSANDRA-3150:
-
Attachment: Screenshot-Hadoop map task list for job_201109212019_1060 on
cassandra01 - Mozilla Firefox.png
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch, Screenshot-Hadoop map task list
> for job_201109212019_1060 on cassandra01 - Mozilla Firefox.png,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3150) ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of whack)
[
https://issues.apache.org/jira/browse/CASSANDRA-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113326#comment-13113326
]
Mck SembWever commented on CASSANDRA-3150:
--
This is persisting to be a problem. And continuously gets worse through the day.
I can see like half my cf being read from one split.
It doesn't matter if it's ByteOrderingPartition or RandomPartitioner.
I've attached two screenshots from hadoop webpages.
> ColumnFormatRecordReader loops forever (StorageService.getSplits(..) out of
> whack)
> --
>
> Key: CASSANDRA-3150
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3150
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
>Affects Versions: 0.8.4
>Reporter: Mck SembWever
>Assignee: Mck SembWever
>Priority: Critical
> Attachments: CASSANDRA-3150.patch,
> attempt_201109071357_0044_m_003040_0.grep-get_range_slices.log
>
>
> From http://thread.gmane.org/gmane.comp.db.cassandra.user/20039
> {quote}
> bq. Cassandra-0.8.4 w/ ByteOrderedPartitioner
> bq. CFIF's inputSplitSize=196608
> bq. 3 map tasks (from 4013) is still running after read 25 million rows.
> bq. Can this be a bug in StorageService.getSplits(..) ?
> getSplits looks pretty foolproof to me but I guess we'd need to add
> more debug logging to rule out a bug there for sure.
> I guess the main alternative would be a bug in the recordreader paging.
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3247) sstableloader ignores option doesn't work correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-3247: Attachment: 3247.patch > sstableloader ignores option doesn't work correctly > --- > > Key: CASSANDRA-3247 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3247 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne >Priority: Minor > Labels: bulkloader > Fix For: 0.8.7, 1.0.0 > > Attachments: 3247.patch > > > The --ignores option is supposed to take an argument but it doesn't. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3247) sstableloader ignores option doesn't work correctly
sstableloader ignores option doesn't work correctly --- Key: CASSANDRA-3247 URL: https://issues.apache.org/jira/browse/CASSANDRA-3247 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Sylvain Lebresne Assignee: Sylvain Lebresne Priority: Minor Fix For: 0.8.7 The --ignores option is supposed to take an argument but it doesn't. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3245) Don't fail when numactl is installed, but NUMA policies are not supported
[
https://issues.apache.org/jira/browse/CASSANDRA-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Schuller updated CASSANDRA-3245:
--
Comment: was deleted
(was: Or we could do something seemingly ugly but probably very effective and
safe: Try running "ls -d / > /dev/null" with numactl and if that fails, assume
it is because numactl isn't working. That should hopefully work in pretty much
any environment.
I'll submit a patch soonish.
)
> Don't fail when numactl is installed, but NUMA policies are not supported
> -
>
> Key: CASSANDRA-3245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3245
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 1.0.0
> Environment: Any Linux system where a 'numactl' executable is
> available, but no NUMA policies are actually supported. EC2 nodes are easy
> examples of environments with no NUMA policy support.
>Reporter: paul cannon
>Assignee: Peter Schuller
>Priority: Minor
> Fix For: 1.0.0
>
>
> When numactl is installed but NUMA policies are not supported, trying to run
> cassandra gives only:
> {noformat}
> numactl: This system does not support NUMA policy
> {noformat}
> ..and the startup script fails there.
> We should probably fail a little more gracefully. Possibly the best way to
> tell if numactl will work is by using:
> {noformat}
> numactl --hardware
> {noformat}
> but I don't have ready access to a machine with proper NUMA support at the
> moment so I can't check how easy it is to tell the difference in the output.
> It looks just as reliable (if possibly a bit more brittle) to check for the
> existence of the directory {{/sys/devices/system/node}}. If that directory
> doesn't exist, we shouldn't even try to use or run numactl.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3245) Don't fail when numactl is installed, but NUMA policies are not supported
[
https://issues.apache.org/jira/browse/CASSANDRA-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113270#comment-13113270
]
Peter Schuller commented on CASSANDRA-3245:
---
Or we could do something seemingly ugly but probably very effective and safe:
Try running "ls -d / > /dev/null" with numactl and if that fails, assume it is
because numactl isn't working. That should hopefully work in pretty much any
environment.
I'll submit a patch soonish.
> Don't fail when numactl is installed, but NUMA policies are not supported
> -
>
> Key: CASSANDRA-3245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3245
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 1.0.0
> Environment: Any Linux system where a 'numactl' executable is
> available, but no NUMA policies are actually supported. EC2 nodes are easy
> examples of environments with no NUMA policy support.
>Reporter: paul cannon
>Assignee: Peter Schuller
>Priority: Minor
> Fix For: 1.0.0
>
>
> When numactl is installed but NUMA policies are not supported, trying to run
> cassandra gives only:
> {noformat}
> numactl: This system does not support NUMA policy
> {noformat}
> ..and the startup script fails there.
> We should probably fail a little more gracefully. Possibly the best way to
> tell if numactl will work is by using:
> {noformat}
> numactl --hardware
> {noformat}
> but I don't have ready access to a machine with proper NUMA support at the
> moment so I can't check how easy it is to tell the difference in the output.
> It looks just as reliable (if possibly a bit more brittle) to check for the
> existence of the directory {{/sys/devices/system/node}}. If that directory
> doesn't exist, we shouldn't even try to use or run numactl.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3245) Don't fail when numactl is installed, but NUMA policies are not supported
[
https://issues.apache.org/jira/browse/CASSANDRA-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113271#comment-13113271
]
Peter Schuller commented on CASSANDRA-3245:
---
Or we could do something seemingly ugly but probably very effective and safe:
Try running "ls -d / > /dev/null" with numactl and if that fails, assume it is
because numactl isn't working. That should hopefully work in pretty much any
environment.
I'll submit a patch soonish.
> Don't fail when numactl is installed, but NUMA policies are not supported
> -
>
> Key: CASSANDRA-3245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3245
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Affects Versions: 1.0.0
> Environment: Any Linux system where a 'numactl' executable is
> available, but no NUMA policies are actually supported. EC2 nodes are easy
> examples of environments with no NUMA policy support.
>Reporter: paul cannon
>Assignee: Peter Schuller
>Priority: Minor
> Fix For: 1.0.0
>
>
> When numactl is installed but NUMA policies are not supported, trying to run
> cassandra gives only:
> {noformat}
> numactl: This system does not support NUMA policy
> {noformat}
> ..and the startup script fails there.
> We should probably fail a little more gracefully. Possibly the best way to
> tell if numactl will work is by using:
> {noformat}
> numactl --hardware
> {noformat}
> but I don't have ready access to a machine with proper NUMA support at the
> moment so I can't check how easy it is to tell the difference in the output.
> It looks just as reliable (if possibly a bit more brittle) to check for the
> existence of the directory {{/sys/devices/system/node}}. If that directory
> doesn't exist, we shouldn't even try to use or run numactl.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several performance problems
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113272#comment-13113272 ] Sylvain Lebresne commented on CASSANDRA-3234: - The patch as is doesn't compile because it removes the import of StringUtils in LeveledManifest.java (it also add the import of DataTracker in AbstractCompactionStrategy which I think is not useful). But other than +1 on 06 v3. > LeveledCompaction has several performance problems > -- > > Key: CASSANDRA-3234 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3234 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis > Fix For: 1.0.0 > > Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, > 0002-add-TrivialOneToOne-optimization.txt, > 0003-fix-leveled-BF-size-calculation.txt, > 0004-avoid-calling-shouldPurge-unless-necessary.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch > > > Two main problems: > - BF size calculation doesn't take into account LCS breaking the output apart > into "bite sized" sstables, so memory use is much higher than predicted > - ManyToMany merging is slow. At least part of this is from running the full > reducer machinery against single input sources, which can be optimized away. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several performance problems
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3234: -- Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch updated to "Sets.difference(L0, sstablesToIgnore).size() + manifest.getLevelCount()" > LeveledCompaction has several performance problems > -- > > Key: CASSANDRA-3234 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3234 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis > Fix For: 1.0.0 > > Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, > 0002-add-TrivialOneToOne-optimization.txt, > 0003-fix-leveled-BF-size-calculation.txt, > 0004-avoid-calling-shouldPurge-unless-necessary.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch > > > Two main problems: > - BF size calculation doesn't take into account LCS breaking the output apart > into "bite sized" sstables, so memory use is much higher than predicted > - ManyToMany merging is slow. At least part of this is from running the full > reducer machinery against single input sources, which can be optimized away. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several performance problems
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3234: -- Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch v2 improves leveled expensiveness equation to L0.size() + count(levels) - ignore.size() > LeveledCompaction has several performance problems > -- > > Key: CASSANDRA-3234 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3234 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis > Fix For: 1.0.0 > > Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, > 0002-add-TrivialOneToOne-optimization.txt, > 0003-fix-leveled-BF-size-calculation.txt, > 0004-avoid-calling-shouldPurge-unless-necessary.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, > 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, > 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, > 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch > > > Two main problems: > - BF size calculation doesn't take into account LCS breaking the output apart > into "bite sized" sstables, so memory use is much higher than predicted > - ManyToMany merging is slow. At least part of this is from running the full > reducer machinery against single input sources, which can be optimized away. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3238) Issue with multi region ec2 and replication updates
[ https://issues.apache.org/jira/browse/CASSANDRA-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3238: -- Component/s: Core Priority: Minor (was: Major) > Issue with multi region ec2 and replication updates > --- > > Key: CASSANDRA-3238 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3238 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Nick Bailey >Assignee: Vijay >Priority: Minor > Fix For: 1.0.0 > > > Using the Ec2MultiRegionSnitch and updating replication settings for a > keyspace seems to cause some issues that require a rolling restart to fix. > The following was observed when updating a keyspace from SimpleStrategy to > NTS in a multi region environment: > * All repairs would hang. Even repairs only against a keyspace that was not > updated. > * Reads at CL.ONE would start to go across region > After a rolling restart of the cluster, repairs started working correctly > again and reads stayed local to the region. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2472) CQL 1.1 and 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-2472. --- Resolution: Fixed Detached the 2.1 tickets and closing this. > CQL 1.1 and 2.0 > --- > > Key: CASSANDRA-2472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2472 > Project: Cassandra > Issue Type: Improvement > Components: API, Core >Reporter: Eric Evans > Labels: cql > Fix For: 1.0.0 > > > Master ticket for issues relating to CQL 1.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2472) CQL 1.1 and 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2472: -- Summary: CQL 1.1 and 2.0 (was: CQL 1.1) > CQL 1.1 and 2.0 > --- > > Key: CASSANDRA-2472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2472 > Project: Cassandra > Issue Type: Improvement > Components: API, Core >Reporter: Eric Evans > Labels: cql > Fix For: 1.0.0 > > > Master ticket for issues relating to CQL 1.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3069) multiget support in CQL (UNION / OR)
[ https://issues.apache.org/jira/browse/CASSANDRA-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3069: -- Issue Type: New Feature (was: Sub-task) Parent: (was: CASSANDRA-2472) > multiget support in CQL (UNION / OR) > > > Key: CASSANDRA-3069 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3069 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 1.0.1 > > Attachments: CASSANDRA-3069.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3069) multiget support in CQL (UNION / OR)
[ https://issues.apache.org/jira/browse/CASSANDRA-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3069: -- Priority: Minor (was: Major) > multiget support in CQL (UNION / OR) > > > Key: CASSANDRA-3069 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3069 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich >Priority: Minor > Fix For: 1.0.1 > > Attachments: CASSANDRA-3069.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2475) Prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2475: -- Priority: Minor (was: Major) Issue Type: New Feature (was: Bug) > Prepared statements > --- > > Key: CASSANDRA-2475 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2475 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Eric Evans >Priority: Minor > Labels: cql > Fix For: 1.0.1 > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2475) Prepared statements
[ https://issues.apache.org/jira/browse/CASSANDRA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2475: -- Issue Type: Bug (was: Sub-task) Parent: (was: CASSANDRA-2472) > Prepared statements > --- > > Key: CASSANDRA-2475 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2475 > Project: Cassandra > Issue Type: Bug > Components: API, Core >Reporter: Eric Evans > Labels: cql > Fix For: 1.0.1 > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2474) CQL support for compound columns
[ https://issues.apache.org/jira/browse/CASSANDRA-2474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-2474: -- Issue Type: New Feature (was: Sub-task) Parent: (was: CASSANDRA-2472) > CQL support for compound columns > > > Key: CASSANDRA-2474 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2474 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Eric Evans >Assignee: Pavel Yaskevich > Labels: cql > Fix For: 1.0.1 > > Attachments: screenshot-1.jpg, screenshot-2.jpg > > > For the most part, this boils down to supporting the specification of > compound column names (the CQL syntax is colon-delimted terms), and then > teaching the decoders (drivers) to create structures from the results. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3197) Separate input and output connection details in ConfigHelper
[ https://issues.apache.org/jira/browse/CASSANDRA-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3197: -- Fix Version/s: (was: 1.0.0) 1.1 This doesn't look quite as trivial as I thought initially, so we should respect the 1.0 freeze and put this in 1.1. > Separate input and output connection details in ConfigHelper > > > Key: CASSANDRA-3197 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3197 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Affects Versions: 0.7.0 >Reporter: Mck SembWever >Assignee: Mck SembWever >Priority: Minor > Fix For: 1.1 > > Attachments: CASSANDRA-3197-extra.patch, CASSANDRA-3197.patch > > > Currently ConfigHelper's getInitialAddress(..) getRpcPort(..) and > getPartitioner(..) > presume CFIF will be using the same cluster as CFOF. > The latter two are a problem for me as on the same servers i'm running two > clusters, one w/ ByteOrderingPartitioner and the other with RP), and i would > like to read from the BOP cluster and write to the RP cluster. > getInitialAddress(..) is of little concern to me. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
