[jira] [Created] (CASSANDRA-11918) NullPointerException in Cleanup Process
vin01 created CASSANDRA-11918: - Summary: NullPointerException in Cleanup Process Key: CASSANDRA-11918 URL: https://issues.apache.org/jira/browse/CASSANDRA-11918 Project: Cassandra Issue Type: Bug Components: Compaction Environment: Cassandra 2.2.4, Centos 6.6 x86_64 Reporter: vin01 Priority: Minor After adding a new node to a 3 node cluster, i started "nodetool cleanup" as recommended. [data-center1] Node1, Node2, Node3 [data-center2] Node4 It finished successfully on one Node4 in the other data-center and took only one minute to finish. On Node3 in data-center1, it threw many exceptions in between :- ERROR [SharedPool-Worker-2] 2016-05-29 00:00:03,571 ErrorMessage.java:336 - Unexpected exception during request java.lang.NullPointerException: null at com.stratio.cassandra.lucene.IndexSearcher.(IndexSearcher.java:77) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at com.stratio.cassandra.lucene.Index.createSecondaryIndexSearcher(Index.java:249) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at org.apache.cassandra.db.index.SecondaryIndexManager.validateIndexSearchersForQuery(SecondaryIndexManager.java:590) ~[apache-cassandra-2.2.4.j ar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.getValidatedIndexExpressions(SelectStatement.java:608) ~[apache-cassandra-2.2.4.jar:2.2. 4] at org.apache.cassandra.cql3.statements.SelectStatement.getRangeCommand(SelectStatement.java:376) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPageableCommand(SelectStatement.java:186) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:172) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:466) ~[apache-cassandra-2.2.4.jar:2.2.4] at com.stratio.cassandra.lucene.IndexQueryHandler.processPrepared(IndexQueryHandler.java:108) ~[cassandra-lucene-index-plugin-2.2.4.0.jar:na] at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:142) ~[apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-2.2.4.jar:2.2.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.2.4.jar:2.2.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.2.4.jar:2.2.4] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] I am not sure what is causing it and is it really a problem. Also if cleanup process is interrupted, do i need to restart the node? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11917) nodetool disablethrift hangs under load
[ https://issues.apache.org/jira/browse/CASSANDRA-11917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated CASSANDRA-11917: Attachment: disable_thrift.txt disable thrift hangs with nodetool > nodetool disablethrift hangs under load > --- > > Key: CASSANDRA-11917 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11917 > Project: Cassandra > Issue Type: Bug >Reporter: Edward Capriolo > Attachments: disable_thrift.txt, stack_summary.txt, > unexpected_throw.txt > > > Under production load some nodetool commands such as disablethrift and drain > never complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11917) nodetool disablethrift hangs under load
[ https://issues.apache.org/jira/browse/CASSANDRA-11917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated CASSANDRA-11917: Attachment: unexpected_throw.txt > nodetool disablethrift hangs under load > --- > > Key: CASSANDRA-11917 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11917 > Project: Cassandra > Issue Type: Bug >Reporter: Edward Capriolo > Attachments: stack_summary.txt, unexpected_throw.txt > > > Under production load some nodetool commands such as disablethrift and drain > never complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11917) nodetool disablethrift hangs under load
[ https://issues.apache.org/jira/browse/CASSANDRA-11917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated CASSANDRA-11917: Attachment: stack_summary.txt A summary of blocked threads > nodetool disablethrift hangs under load > --- > > Key: CASSANDRA-11917 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11917 > Project: Cassandra > Issue Type: Bug >Reporter: Edward Capriolo > Attachments: stack_summary.txt > > > Under production load some nodetool commands such as disablethrift and drain > never complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11917) nodetool disablethrift hangs under load
Edward Capriolo created CASSANDRA-11917: --- Summary: nodetool disablethrift hangs under load Key: CASSANDRA-11917 URL: https://issues.apache.org/jira/browse/CASSANDRA-11917 Project: Cassandra Issue Type: Bug Reporter: Edward Capriolo Under production load some nodetool commands such as disablethrift and drain never complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11916) Exception In Compaction Executor - java.lang.IllegalArgumentException: null
Bhuvan Rawal created CASSANDRA-11916: Summary: Exception In Compaction Executor - java.lang.IllegalArgumentException: null Key: CASSANDRA-11916 URL: https://issues.apache.org/jira/browse/CASSANDRA-11916 Project: Cassandra Issue Type: Bug Components: Compaction Reporter: Bhuvan Rawal Priority: Critical Fix For: 3.0.x We are using Cassandra 3.0.3 with Level ordered compaction strategy with default compression. While doing some load tests, I can observe these messages after near fixed intervals of 15-20 seconds each on just one node amongst 6 node cluster: ERROR [CompactionExecutor:23] 2016-05-29 01:29:42,643 CassandraDaemon.java:195 - Exception in thread Thread[CompactionExecutor:23,1,main] java.lang.IllegalArgumentException: null at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_45] at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:114) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:183) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:302) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:181) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$200(BigTableScanner.java:51) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:280) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.0.3.jar:3.0.3] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.3.jar:3.0.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11915) [PATCH] doc: correct section number of opcode in protocol spec
Amos Jianjun Kong created CASSANDRA-11915: - Summary: [PATCH] doc: correct section number of opcode in protocol spec Key: CASSANDRA-11915 URL: https://issues.apache.org/jira/browse/CASSANDRA-11915 Project: Cassandra Issue Type: Bug Components: Documentation and Website Environment: problem doesn't related with environment. MacBook Reporter: Amos Jianjun Kong Fix For: 3.8 Attachments: 0001-doc-correct-section-number-of-opcode-in-protocol-spe.patch This patch just corrected the section number, opcode detail is described in Section 2.4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305295#comment-15305295
]
Kaide Mu commented on CASSANDRA-11875:
--
Just unified things into SSTableConverter and StandaloneConverter.
Also added a new Test suit SSTableConversionTest, but it has some problem when
performing conversion.
{{ERROR 10:39:13 Attempting to load already loaded table
SSTableRewriterTest.Standard1}}
[Patch|https://github.com/kdmu/cassandra/commit/e8637a531e874310a49bc5dff71c94b8841c2e09]
(pushed to trunk instead of a temporary working branch, will fix it)
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11711) testJsonThreadSafety is failing / flapping
[
https://issues.apache.org/jira/browse/CASSANDRA-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-11711:
Status: Ready to Commit (was: Patch Available)
> testJsonThreadSafety is failing / flapping
> --
>
> Key: CASSANDRA-11711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11711
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Alex Petrov
>Assignee: Tyler Hobbs
>Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> {{JsonTest::testJsonThreadSafety}} is failing quite often recently:
> https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-11540-2.2-testall/lastCompletedBuild/testReport/org.apache.cassandra.cql3.validation.entities/JsonTest/testJsonThreadSafety/
> Output looks like
> {code}
> Stacktrace
> java.util.concurrent.TimeoutException
> at java.util.concurrent.FutureTask.get(FutureTask.java:201)
> at
> org.apache.cassandra.cql3.validation.entities.JsonTest.testJsonThreadSafety(JsonTest.java:1028)
> WARN 12:19:23 Small commitlog volume detected at
> build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to
> 1982. You can override this in cassandra.yaml
> WARN 12:19:23 Small commitlog volume detected at
> build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to
> 1982. You can override this in cassandra.yaml
> WARN 12:19:23 Only 5581 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> WARN 12:19:23 Only 5581 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> Seed 889742091470
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11711) testJsonThreadSafety is failing / flapping
[
https://issues.apache.org/jira/browse/CASSANDRA-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305228#comment-15305228
]
Alex Petrov commented on CASSANDRA-11711:
-
I agree that it looks rather like the test and code underneath is correct and
the error (in my experience occurs more when there're many concurrent jobs
running, which results into timeouts). I also couldn't reproduce it locally no
matter how many times I ran it, increasing number of iterations and so didn't
break it, too.
I've triggered a job which ran a build several times and all runs of this test
were clean.
+1
> testJsonThreadSafety is failing / flapping
> --
>
> Key: CASSANDRA-11711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11711
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Alex Petrov
>Assignee: Tyler Hobbs
>Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> {{JsonTest::testJsonThreadSafety}} is failing quite often recently:
> https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-11540-2.2-testall/lastCompletedBuild/testReport/org.apache.cassandra.cql3.validation.entities/JsonTest/testJsonThreadSafety/
> Output looks like
> {code}
> Stacktrace
> java.util.concurrent.TimeoutException
> at java.util.concurrent.FutureTask.get(FutureTask.java:201)
> at
> org.apache.cassandra.cql3.validation.entities.JsonTest.testJsonThreadSafety(JsonTest.java:1028)
> WARN 12:19:23 Small commitlog volume detected at
> build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to
> 1982. You can override this in cassandra.yaml
> WARN 12:19:23 Small commitlog volume detected at
> build/test/cassandra/commitlog:30; setting commitlog_total_space_in_mb to
> 1982. You can override this in cassandra.yaml
> WARN 12:19:23 Only 5581 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> WARN 12:19:23 Only 5581 MB free across all data volumes. Consider adding
> more capacity to your cluster or removing obsolete snapshots
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> WARN 12:19:26 Aggregation query used without partition key
> Seed 889742091470
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
