[jira] [Resolved] (CASSANDRA-10170) Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to upgrade with java.lang.ClassCastException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-10170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Darla Baker resolved CASSANDRA-10170.
-
Resolution: Not A Problem
Following testing on the -jb- files we have concluded that the issue observed
was caused by corrupt -jb- tables that pre-existed the upgrade effort.
> Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to
> upgrade with java.lang.ClassCastException: null
>
>
> Key: CASSANDRA-10170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10170
> Project: Cassandra
> Issue Type: Bug
>Reporter: Darla Baker
>Priority: Critical
>
> Not all -jb- tables throw this error, but I have a set of tables that will
> reproduce the error:
> {code}
> Error upgrading
> SSTableReader(path='/var/lib/cassandra/data//-13fdf0a04a9511e59ebb3130028babc8/--jb-2-Data.db'):
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> java.lang.ClassCastException:
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
> at
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
> at org.apache.cassandra.db.compaction.Upgrader.upgrade(Upgrader.java:89)
> at
> org.apache.cassandra.tools.StandaloneUpgrader.main(StandaloneUpgrader.java:104)
> {code}
> This was seen in CASSANDRA-6738 (Not Reproduced), CASSANDRA-7112 (Fixed 2.1
> rc1) and CASSANDRA-7990 (Fixed 2.1.1). I decided to open a new Jira since
> this is being observed in a version where it should have been fixed and
> wasn't sure if perhaps one of the other Jiras should be reopened instead so
> I'll leave it to the experts.
> Since the sstables are sensitive, I can share the data privately as needed to
> reproduce and diagnose the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10170) Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to upgrade with java.lang.ClassCastException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-10170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14713563#comment-14713563
]
Darla Baker commented on CASSANDRA-10170:
-
Loaded another set of -jb- files from another table that is also halting the
upgrade and received a different error:
{code}
ERROR [Thrift:16] 2015-08-26 10:40:43,375 CustomTThreadPoolServer.java (line
219) Error occurred during processing of message.
java.lang.ArrayIndexOutOfBoundsException: 3
at
org.apache.cassandra.cql3.statements.ColumnGroupMap.add(ColumnGroupMap.java:48)
at
org.apache.cassandra.cql3.statements.ColumnGroupMap.access$200(ColumnGroupMap.java:32)
at
org.apache.cassandra.cql3.statements.ColumnGroupMap$Builder.add(ColumnGroupMap.java:140)
at
org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1174)
at
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1077)
at
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:283)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:260)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:63)
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler.statementExecution(DseQueryHandler.java:208)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:88)
at
org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
at
com.datastax.bdp.server.DseServer.execute_cql3_query(DseServer.java:568)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to
> upgrade with java.lang.ClassCastException: null
>
>
> Key: CASSANDRA-10170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10170
> Project: Cassandra
> Issue Type: Bug
>Reporter: Darla Baker
>Priority: Critical
>
> Not all -jb- tables throw this error, but I have a set of tables that will
> reproduce the error:
> {code}
> Error upgrading
> SSTableReader(path='/var/lib/cassandra/data//-13fdf0a04a9511e59ebb3130028babc8/--jb-2-Data.db'):
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> java.lang.ClassCastException:
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
> at
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121
[jira] [Commented] (CASSANDRA-10170) Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to upgrade with java.lang.ClassCastException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-10170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711478#comment-14711478
]
Darla Baker commented on CASSANDRA-10170:
-
I loaded the -jb- data sstable into a fresh install of c* 2.0.10 using
sstableloader and it loaded fine. But when running select * from .
in cqlsh I get the following stack trace:
{code}
CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:97,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.filter.ColumnCounter$GroupByPrefix.count(ColumnCounter.java:117)
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:192)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:101)
at
org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:75)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1594)
at
org.apache.cassandra.db.ColumnFamilyStore$9.computeNext(ColumnFamilyStore.java:1590)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1750)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709)
at
org.apache.cassandra.db.RangeSliceCommand.executeLocally(RangeSliceCommand.java:137)
at
org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1394)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to
> upgrade with java.lang.ClassCastException: null
>
>
> Key: CASSANDRA-10170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10170
> Project: Cassandra
> Issue Type: Bug
>Reporter: Darla Baker
>Priority: Critical
>
> Not all -jb- tables throw this error, but I have a set of tables that will
> reproduce the error:
> {code}
> Error upgrading
> SSTableReader(path='/var/lib/cassandra/data//-13fdf0a04a9511e59ebb3130028babc8/--jb-2-Data.db'):
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> java.lang.ClassCastException:
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
> at
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
> at
> org.apache.c
[jira] [Updated] (CASSANDRA-10170) Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to upgrade with java.lang.ClassCastException: null
[
https://issues.apache.org/jira/browse/CASSANDRA-10170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Darla Baker updated CASSANDRA-10170:
Summary: Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are
refusing to upgrade with java.lang.ClassCastException: null (was:
Upgradesstables from 1.2.x -jb- to 2.1.x -ka- some files are refusing to
upgrade with java.lang.ClassCastException: null)
> Upgradesstables from 2.0.8 -jb- to 2.1.x -ka- some files are refusing to
> upgrade with java.lang.ClassCastException: null
>
>
> Key: CASSANDRA-10170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10170
> Project: Cassandra
> Issue Type: Bug
>Reporter: Darla Baker
>Priority: Critical
>
> Not all -jb- tables throw this error, but I have a set of tables that will
> reproduce the error:
> {code}
> Error upgrading
> SSTableReader(path='/var/lib/cassandra/data//-13fdf0a04a9511e59ebb3130028babc8/--jb-2-Data.db'):
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> java.lang.ClassCastException:
> org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
> org.apache.cassandra.db.composites.CellName
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
> at
> org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
> at
> org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
> at org.apache.cassandra.db.compaction.Upgrader.upgrade(Upgrader.java:89)
> at
> org.apache.cassandra.tools.StandaloneUpgrader.main(StandaloneUpgrader.java:104)
> {code}
> This was seen in CASSANDRA-6738 (Not Reproduced), CASSANDRA-7112 (Fixed 2.1
> rc1) and CASSANDRA-7990 (Fixed 2.1.1). I decided to open a new Jira since
> this is being observed in a version where it should have been fixed and
> wasn't sure if perhaps one of the other Jiras should be reopened instead so
> I'll leave it to the experts.
> Since the sstables are sensitive, I can share the data privately as needed to
> reproduce and diagnose the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10170) Upgradesstables from 1.2.x -jb- to 2.1.x -ka- some files are refusing to upgrade with java.lang.ClassCastException: null
Darla Baker created CASSANDRA-10170:
---
Summary: Upgradesstables from 1.2.x -jb- to 2.1.x -ka- some files
are refusing to upgrade with java.lang.ClassCastException: null
Key: CASSANDRA-10170
URL: https://issues.apache.org/jira/browse/CASSANDRA-10170
Project: Cassandra
Issue Type: Bug
Reporter: Darla Baker
Priority: Critical
Not all -jb- tables throw this error, but I have a set of tables that will
reproduce the error:
{code}
Error upgrading
SSTableReader(path='/var/lib/cassandra/data//-13fdf0a04a9511e59ebb3130028babc8/--jb-2-Data.db'):
org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
org.apache.cassandra.db.composites.CellName
java.lang.ClassCastException:
org.apache.cassandra.db.composites.CompoundComposite cannot be cast to
org.apache.cassandra.db.composites.CellName
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86)
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
at org.apache.cassandra.db.compaction.Upgrader.upgrade(Upgrader.java:89)
at
org.apache.cassandra.tools.StandaloneUpgrader.main(StandaloneUpgrader.java:104)
{code}
This was seen in CASSANDRA-6738 (Not Reproduced), CASSANDRA-7112 (Fixed 2.1
rc1) and CASSANDRA-7990 (Fixed 2.1.1). I decided to open a new Jira since this
is being observed in a version where it should have been fixed and wasn't sure
if perhaps one of the other Jiras should be reopened instead so I'll leave it
to the experts.
Since the sstables are sensitive, I can share the data privately as needed to
reproduce and diagnose the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10133) Make updates to system_auth invalidate permission_validity cache
Darla Baker created CASSANDRA-10133: --- Summary: Make updates to system_auth invalidate permission_validity cache Key: CASSANDRA-10133 URL: https://issues.apache.org/jira/browse/CASSANDRA-10133 Project: Cassandra Issue Type: New Feature Reporter: Darla Baker Currently a change to system_auth (password, add/remove user) will not take effect until the permission_validity_in_ms time expires or the nodes in the cluster are recycled. In larger clusters this setting can be rather long so changes don't take effect as quickly as desired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9126) java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey
[
https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600119#comment-14600119
]
Darla Baker commented on CASSANDRA-9126:
Another user reported this same Error in 2.0.9 and they are using hsha and lcs.
scrub fixes the issue but it returns on occasion. As I get more details, I'll
update.
> java.lang.RuntimeException: Last written key DecoratedKey >= current key
> DecoratedKey
> -
>
> Key: CASSANDRA-9126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: srinivasu gottipati
>Priority: Critical
> Fix For: 2.0.x
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue
> was raised in earlier versions and marked as closed. However it still appears
> in 2.0.14). In our case, compaction is not getting succeeded and keep failing
> with this error.:
> {code}java.lang.RuntimeException: Last written key
> DecoratedKey(3462767860784856708,
> 354038323137333038305f3330325f31355f474d4543454f) >= current key
> DecoratedKey(3462334604624154281,
> 354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> ...
> Stacktrace:{code}
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9078) Fail repair when cluster is in a mixed version state
Darla Baker created CASSANDRA-9078: -- Summary: Fail repair when cluster is in a mixed version state Key: CASSANDRA-9078 URL: https://issues.apache.org/jira/browse/CASSANDRA-9078 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Darla Baker Priority: Minor Users often have cron jobs or other automation set up to perform regular repair maintenance in the cluster. During an upgrade when the cluster is in a mixed state this can be problematic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7280) Hadoop support not respecting cassandra.input.split.size
[ https://issues.apache.org/jira/browse/CASSANDRA-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165161#comment-14165161 ] Darla Baker commented on CASSANDRA-7280: Any chance this will be assigned soon? > Hadoop support not respecting cassandra.input.split.size > > > Key: CASSANDRA-7280 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7280 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Jeremy Hanna > > Long ago (0.7), I tried to set the cassandra.input.split.size property and > never really got it to respect that property. However the batch size was > useful for what I needed to affect the timeouts. > Now with the cql record reader and the native paging, users can specify > queries potentially using allow filtering clauses. The input split size is > more important because the server may have to scan through many many records > to get matching records. If the user can effectively set the input split > size, then that gives a hard limit on how many records it will traverse. > Currently it appears to be overriding the property, perhaps in the > client.describe_splits_ex method on the server side. > It can be argued that users shouldn't be using allow filtering clauses in > their cql in the first place. However it is still a bug that the input split > size is not honored. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-7552) Compactions Pending build up when using LCS
Darla Baker created CASSANDRA-7552: -- Summary: Compactions Pending build up when using LCS Key: CASSANDRA-7552 URL: https://issues.apache.org/jira/browse/CASSANDRA-7552 Project: Cassandra Issue Type: Bug Components: Core Reporter: Darla Baker We seem to be hitting an issue with LeveledCompactionStrategy while running performance tests on a 4 node cassandra installation. We are currently using Cassandra 2.0.7. In summary, we run a tests consisting of approximatively, 8000 inserts/sec, 16,000 gets/sec, and 8,000 deletes/sec. We have a grace period of 12 hours on our column families. At this rate, we observe a stable pending compaction tasks for about 22 to 26 hours. After that period, something happens and the pending compaction tasks starts to increase rapidly, sometimes on one or two servers, but sometimes on all four of them. This goes on until the uncompacted SStables start consuming all the disk space, after which the cassandra cluster generally fails. When this occurs, the Compaction completed tasks rate is usually reducing over time, which seems to indicate that it takes more and more time to run the existing compaction tasks. At different occasions, I can reproduce a similar issue in less than 12 hours. While the traffic rate remains constant, we seem to be hitting this at various intervals. Yesterday I could reproduce in less than 6 hours. We have two different deployments on which we have tested this issue: 1. 4x IBM HS22, using RAMDISK as cassandra data directory (thus eliminating disk I/O) 2. 8x IBM HS23, with SSD disks, deployed in two "geo-redundant" data centers of 4 nodes each, and a latency of 50ms between the data centers. I can reproduce the "compaction tasks falling behind" on both these setup, although they could be occurring for different reasons. Because of #1, I do not believe we are hitting an I/O bottleneck just yet. As an additional interesting node, if I artificially pause the traffic when I see the pending compaction task issue occurring, then: 1. The pending compaction tasks obviously stops to increase, but stay at the same number for 15 minutes (as if nothing is running). 2. The completed compaction tasks falls to 0 for 15 minutes 3. After 15 to 20 minutes, out of the blue, all compaction completes in less than 2 minutes. If I restart the traffic after that, the system is stable for a few hours, but the issue always comes back. We have written a small test tool that reproduce our application's Cassandra interaction. We have not successfully run a test for more than 30 hours under load, and every failure after that time would follow a similar pattern. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-6972) Throw an ERROR when auto_bootstrap: true and bootstrapping node is listed in seeds
Darla Baker created CASSANDRA-6972: -- Summary: Throw an ERROR when auto_bootstrap: true and bootstrapping node is listed in seeds Key: CASSANDRA-6972 URL: https://issues.apache.org/jira/browse/CASSANDRA-6972 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Darla Baker Priority: Minor Obviously when this condition exists the node will not bootstrap. But it is not obvious from the logs why it is not bootstrapping. Throwing an error would make it obvious and therefore faster to correct. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-6889) Add the ability to use a url as the input to the --file parameter of cqlsh
Darla Baker created CASSANDRA-6889: -- Summary: Add the ability to use a url as the input to the --file parameter of cqlsh Key: CASSANDRA-6889 URL: https://issues.apache.org/jira/browse/CASSANDRA-6889 Project: Cassandra Issue Type: Improvement Reporter: Darla Baker Priority: Minor Customer request to use a gist or similar url pointing to a cqlsh script rather than a local file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-6170) Modify AbstractCassandraDaemon.initLog4j() to allow for hotfixing log level on on a cassandra class
Darla Baker created CASSANDRA-6170: -- Summary: Modify AbstractCassandraDaemon.initLog4j() to allow for hotfixing log level on on a cassandra class Key: CASSANDRA-6170 URL: https://issues.apache.org/jira/browse/CASSANDRA-6170 Project: Cassandra Issue Type: New Feature Components: Config Reporter: Darla Baker When customer wants to bump up log level of a cassandra class, here is the procedure they follow: # Add the class name and log level to log4j-server.properties into a revision identified directory. # The new directory is then symbolically linked to the location where cassandra looks for that directory. However cassandra and it's log4j continue to watch the old location The reason for this AbstractCassandraDaemon.initLog4j() uses this configFileName = new File(configLocation.toURI()).getCanonicalPath(); The customer believes if we change that to the following, this will allow the symlink to work properly. configFileName = new File(configLocation.toURI()).getPath(); If it were possible to add a configuration that would invoke this change on "True" that would be ideal. Or find another method to allow hotfixing the log4j changes. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6127) vnodes don't scale to hundreds of nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783481#comment-13783481 ] Darla Baker commented on CASSANDRA-6127: Per Jonathan's request, I'm adding an update here regarding eBay's experience on https://support.datastax.com/tickets/6928 which was the result of first stage of executing the plan from https://support.datastax.com/requests/6636. He had an existing 32 cluster DSE 3.1.0 cluster in their PHX data center. Their plan was to add a second data center to the cluster in SLC with 50 nodes and vnodes enabled. They were to begin with bringing all nodes up with auto bootstrapping turned off to prevent any data streaming until they were ready to make other changes to bring the data center fully online. Essentially immediately upon bringing the nodes up in SLC, the nodes in PHX began reporting as down and he began receiving SMS messages and calls from application engineers that the application which uses that cluster was down. As we were in triage mode, the most expedient course of action was to shut down the SLC nodes and remove them from gossip. Upon trying to execute the nodetool removenode command we hit CASSANDRA-5857 although we thought up to this point that nodetool decommission was responsible for the issue. In any case, we started the process of executing the workaround as per that ticket. At the point we parted, the process was going slowly but he reported it was working and the nodes were disappearing from the ring and the application engineers were reporting that the application was back online. At some point during the weekend, Alex reached out to Jeremy who was on call and Jeremy who was able to finally get the nodes removed from gossip and fully stabilize the 32 node PHX data center and fully decommission the SLC data center. Alex attached some logs to the ticket during the event. We were seeing node flapping and NPEs during the event. Ticket https://support.datastax.com/tickets/6917 contains some additional details on the test cases. Ticket https://support.datastax.com/tickets/6939 contains the alternate plan that eBay is considering in light of the difficulties encountered with bringing SLC online. > vnodes don't scale to hundreds of nodes > --- > > Key: CASSANDRA-6127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6127 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Any cluster that has vnodes and consists of hundreds of > physical nodes. >Reporter: Tupshin Harper >Assignee: Jonathan Ellis > > There are a lot of gossip-related issues related to very wide clusters that > also have vnodes enabled. Let's use this ticket as a master in case there are > sub-tickets. > The most obvious symptom I've seen is with 1000 nodes in EC2 with m1.xlarge > instances. Each node configured with 32 vnodes. > Without vnodes, cluster spins up fine and is ready to handle requests within > 30 minutes or less. > With vnodes, nodes are reporting constant up/down flapping messages with no > external load on the cluster. After a couple of hours, they were still > flapping, had very high cpu load, and the cluster never looked like it was > going to stabilize or be useful for traffic. -- This message was sent by Atlassian JIRA (v6.1#6144)
