[jira] [Commented] (CASSANDRA-10855) Use Caffeine (W-TinyLFU) for on-heap caches
[ https://issues.apache.org/jira/browse/CASSANDRA-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381078#comment-15381078 ] Ben Manes commented on CASSANDRA-10855: --- Any remaining blockers? > Use Caffeine (W-TinyLFU) for on-heap caches > --- > > Key: CASSANDRA-10855 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10855 > Project: Cassandra > Issue Type: Improvement >Reporter: Ben Manes > Labels: performance > > Cassandra currently uses > [ConcurrentLinkedHashMap|https://code.google.com/p/concurrentlinkedhashmap] > for performance critical caches (key, counter) and Guava's cache for > non-critical (auth, metrics, security). All of these usages have been > replaced by [Caffeine|https://github.com/ben-manes/caffeine], written by the > author of the previously mentioned libraries. > The primary incentive is to switch from LRU policy to W-TinyLFU, which > provides [near optimal|https://github.com/ben-manes/caffeine/wiki/Efficiency] > hit rates. It performs particularly well in database and search traces, is > scan resistant, and as adds a very small time/space overhead to LRU. > Secondarily, Guava's caches never obtained similar > [performance|https://github.com/ben-manes/caffeine/wiki/Benchmarks] to CLHM > due to some optimizations not being ported over. This change results in > faster reads and not creating garbage as a side-effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12194) dtest failure in upgrade_tests.cql_tests.TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x.compact_metadata_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov resolved CASSANDRA-12194.
-
Resolution: Fixed
Fixed by the [CASSANDRA-12193].
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x.compact_metadata_test
>
>
> Key: CASSANDRA-12194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12194
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
>Assignee: Alex Petrov
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/compact_metadata_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 2636, in compact_metadata_test
> assert_one(cursor, "SELECT * FROM bar", [1, 2])
> File "/home/automaton/cassandra-dtest/assertions.py", line 123, in
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got
> {}".format([expected], query, list_res)
> "Expected [[1, 2]] from SELECT * FROM bar, but got [[1, None]]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/2] cassandra git commit: imports cleanup
Repository: cassandra
Updated Branches:
refs/heads/trunk f6ca48251 -> 087264f29
http://git-wip-us.apache.org/repos/asf/cassandra/blob/087264f2/src/java/org/apache/cassandra/net/MessageIn.java
--
diff --git a/src/java/org/apache/cassandra/net/MessageIn.java
b/src/java/org/apache/cassandra/net/MessageIn.java
index 014ee93..fe85595 100644
--- a/src/java/org/apache/cassandra/net/MessageIn.java
+++ b/src/java/org/apache/cassandra/net/MessageIn.java
@@ -27,10 +27,8 @@ import com.google.common.collect.ImmutableMap;
import org.apache.cassandra.concurrent.Stage;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.monitoring.ConstructionTime;
-import org.apache.cassandra.db.monitoring.MonitorableImpl;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataInputPlus;
-import org.apache.cassandra.io.util.FileUtils;
public class MessageIn
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/087264f2/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
b/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
index 312daed..52625bf 100644
--- a/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
+++ b/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java
@@ -21,7 +21,6 @@ import java.net.InetAddress;
import java.util.*;
import com.google.common.base.Predicate;
-import com.google.common.collect.Sets;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.common.util.concurrent.MoreExecutors;
import org.slf4j.Logger;
@@ -30,8 +29,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.compaction.CompactionManager;
import org.apache.cassandra.dht.Bounds;
-import org.apache.cassandra.dht.Range;
-import org.apache.cassandra.dht.Token;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.MessageIn;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/087264f2/src/java/org/apache/cassandra/repair/ValidationTask.java
--
diff --git a/src/java/org/apache/cassandra/repair/ValidationTask.java
b/src/java/org/apache/cassandra/repair/ValidationTask.java
index bd866d2..177ad3e 100644
--- a/src/java/org/apache/cassandra/repair/ValidationTask.java
+++ b/src/java/org/apache/cassandra/repair/ValidationTask.java
@@ -18,16 +18,12 @@
package org.apache.cassandra.repair;
import java.net.InetAddress;
-import java.util.Map;
import com.google.common.util.concurrent.AbstractFuture;
-import org.apache.cassandra.dht.Range;
-import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.RepairException;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.repair.messages.ValidationRequest;
-import org.apache.cassandra.utils.MerkleTree;
import org.apache.cassandra.utils.MerkleTrees;
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/087264f2/src/java/org/apache/cassandra/scheduler/WeightedQueue.java
--
diff --git a/src/java/org/apache/cassandra/scheduler/WeightedQueue.java
b/src/java/org/apache/cassandra/scheduler/WeightedQueue.java
index 298938d..76c7e9d 100644
--- a/src/java/org/apache/cassandra/scheduler/WeightedQueue.java
+++ b/src/java/org/apache/cassandra/scheduler/WeightedQueue.java
@@ -20,9 +20,6 @@ package org.apache.cassandra.scheduler;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.TimeUnit;
-import java.lang.management.ManagementFactory;
-import javax.management.MBeanServer;
-import javax.management.ObjectName;
import org.apache.cassandra.metrics.LatencyMetrics;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/087264f2/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
--
diff --git a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
index 93591f0..c70161e 100644
--- a/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
+++ b/src/java/org/apache/cassandra/schema/LegacySchemaMigrator.java
@@ -40,9 +40,7 @@ import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.db.rows.RowIterator;
import org.apache.cassandra.db.rows.UnfilteredRowIterators;
import org.apache.cassandra.exceptions.InvalidRequestException;
-import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
-import org.apache.cassandra.utils.c
[2/2] cassandra git commit: imports cleanup
imports cleanup Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/087264f2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/087264f2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/087264f2 Branch: refs/heads/trunk Commit: 087264f29c309ed0b2a18cc20b79016a05847f1c Parents: f6ca482 Author: Dave Brosius Authored: Sat Jul 16 12:40:05 2016 -0400 Committer: Dave Brosius Committed: Sat Jul 16 12:40:05 2016 -0400 -- .../cassandra/thrift/AuthenticationException.java | 13 - .../org/apache/cassandra/thrift/CfSplit.java | 13 - .../org/apache/cassandra/thrift/Column.java| 11 --- .../org/apache/cassandra/thrift/ColumnDef.java | 12 .../cassandra/thrift/ColumnOrSuperColumn.java | 13 - .../org/apache/cassandra/thrift/ColumnPath.java| 12 .../org/apache/cassandra/thrift/CounterColumn.java | 1 - .../apache/cassandra/auth/CassandraAuthorizer.java | 6 -- .../org/apache/cassandra/auth/RoleResource.java| 1 - .../cassandra/auth/jmx/AuthorizationProxy.java | 3 --- .../org/apache/cassandra/cache/ChunkCache.java | 1 - .../AbstractLocalAwareExecutorService.java | 2 -- .../concurrent/DebuggableThreadPoolExecutor.java | 3 --- .../apache/cassandra/concurrent/ExecutorLocal.java | 1 - .../apache/cassandra/concurrent/StageManager.java | 1 - .../cassandra/config/ParameterizedClass.java | 1 - .../org/apache/cassandra/cql3/AbstractMarker.java | 1 - src/java/org/apache/cassandra/cql3/Attributes.java | 3 --- .../apache/cassandra/cql3/ColumnIdentifier.java| 6 -- .../org/apache/cassandra/cql3/FieldIdentifier.java | 1 - src/java/org/apache/cassandra/cql3/Json.java | 1 - src/java/org/apache/cassandra/cql3/Operation.java | 1 - src/java/org/apache/cassandra/cql3/Tuples.java | 1 - .../cassandra/cql3/functions/AggregateFcts.java| 1 - .../apache/cassandra/cql3/functions/TimeFcts.java | 1 - .../apache/cassandra/cql3/functions/ToJsonFct.java | 2 -- .../cassandra/cql3/selection/FieldSelector.java| 1 - .../cassandra/cql3/selection/Selectable.java | 1 - .../apache/cassandra/cql3/selection/Selection.java | 1 - .../cql3/selection/SelectionColumnMapping.java | 2 -- .../cassandra/cql3/selection/TermSelector.java | 3 --- .../cql3/statements/AlterTableStatementColumn.java | 1 - .../cql3/statements/CreateTypeStatement.java | 2 -- .../cql3/statements/CreateViewStatement.java | 1 - .../cql3/statements/DropViewStatement.java | 2 -- .../cassandra/cql3/statements/IndexTarget.java | 1 - .../cql3/statements/ModificationStatement.java | 1 - .../cassandra/cql3/statements/SelectStatement.java | 4 .../cql3/statements/TruncateStatement.java | 1 - .../org/apache/cassandra/db/BufferClustering.java | 4 .../org/apache/cassandra/db/CounterMutation.java | 1 - .../cassandra/db/CounterMutationVerbHandler.java | 1 - src/java/org/apache/cassandra/db/LivenessInfo.java | 1 - .../apache/cassandra/db/MutableDeletionInfo.java | 1 - .../apache/cassandra/db/MutationVerbHandler.java | 1 - .../apache/cassandra/db/NativeDecoratedKey.java| 1 - .../org/apache/cassandra/db/PartitionPosition.java | 1 - .../cassandra/db/PartitionRangeReadCommand.java| 1 - .../cassandra/db/SinglePartitionReadCommand.java | 1 - src/java/org/apache/cassandra/db/Slices.java | 1 - .../cassandra/db/commitlog/CommitLogReader.java| 1 - .../cassandra/db/commitlog/CompressedSegment.java | 2 -- .../db/compaction/AbstractCompactionStrategy.java | 1 - .../db/compaction/CompactionController.java| 1 - .../db/compaction/CompactionManagerMBean.java | 1 - .../db/compaction/LeveledCompactionStrategy.java | 1 - .../cassandra/db/compaction/LeveledManifest.java | 1 - .../compaction/SizeTieredCompactionStrategy.java | 1 - .../apache/cassandra/db/filter/ColumnFilter.java | 1 - .../org/apache/cassandra/db/lifecycle/View.java| 2 -- .../org/apache/cassandra/db/marshal/TimeType.java | 1 - .../org/apache/cassandra/db/marshal/UserType.java | 2 -- .../cassandra/db/partitions/PartitionIterator.java | 2 -- .../db/partitions/PartitionIterators.java | 1 - .../org/apache/cassandra/db/rows/CellPath.java | 1 - .../cassandra/db/rows/UnfilteredRowIterator.java | 3 --- .../cassandra/db/rows/WithOnlyQueriedData.java | 1 - .../org/apache/cassandra/db/view/ViewBuilder.java | 2 -- .../cassandra/db/view/ViewUpdateGenerator.java | 1 - src/java/org/apache/cassandra/dht/Bounds.java | 1 - .../org/apache/cassandra/hints/HintsBuffer.java| 1 - .../cassandra/hints/HintsDispatchTrigger.java | 1 - .../cassandra/index/internal/IndexEntry.java | 1 - .
[jira] [Commented] (CASSANDRA-9608) Support Java 9
[
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380816#comment-15380816
]
Robert Stupp commented on CASSANDRA-9608:
-
Updated the branch so far with patched versions of jamm and ohc. Seems many
more utests work now.
The [commit for
jamm|https://github.com/snazy/jamm/commit/c81473ebf3cf41bcc5e7ebfeb3559bc88249a486]
feels quite hacky - so I think that needs some refurbishing.
The [commit for
ohc|https://github.com/snazy/ohc/commit/33889bba38cdc8089be37a007d30646c9771ad6c]
just adds support to parse Java 9 EA version numbers.
UDFs (both Java + JavaScript) don't work yet - all unit tests using UDFs fail
(did not investigate yet).
Some cipher tests and an mmap test fail, too. Some utests still fail with
abnormal JVM exit (probably a missing jigsaw module).
> Support Java 9
> --
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
> Issue Type: Task
>Reporter: Robert Stupp
>Priority: Minor
>
> This ticket is intended to group all issues found to support Java 9 in the
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved.
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods
> {{monitorEnter}} + {{monitorExit}}. These methods are used by
> {{o.a.c.utils.concurrent.Locks}} which is only used by
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early
> development phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12181) Include table name in "Cannot get comparator" exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380625#comment-15380625 ] Robert Stupp commented on CASSANDRA-12181: -- Yes, would be great. > Include table name in "Cannot get comparator" exception > --- > > Key: CASSANDRA-12181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12181 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Trivial > Attachments: CASSANDRA-12181_3.0.txt > > > Having table name will help in debugging the following exception. > ERROR [MutationStage:xx] CassandraDaemon.java (line 199) Exception in thread > Thread[MutationStage:3788,5,main] > clusterName=itms8shared20 > java.lang.RuntimeException: Cannot get comparator 2 in > org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type). > > This might be due to a mismatch between the schema and the data read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12181) Include table name in "Cannot get comparator" exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380623#comment-15380623 ] sankalp kohli edited comment on CASSANDRA-12181 at 7/16/16 8:25 AM: Sure +1 Let me know if you need a modified patch was (Author: kohlisankalp): Sure +1 > Include table name in "Cannot get comparator" exception > --- > > Key: CASSANDRA-12181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12181 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Trivial > Attachments: CASSANDRA-12181_3.0.txt > > > Having table name will help in debugging the following exception. > ERROR [MutationStage:xx] CassandraDaemon.java (line 199) Exception in thread > Thread[MutationStage:3788,5,main] > clusterName=itms8shared20 > java.lang.RuntimeException: Cannot get comparator 2 in > org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type). > > This might be due to a mismatch between the schema and the data read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12181) Include table name in "Cannot get comparator" exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380623#comment-15380623 ] sankalp kohli commented on CASSANDRA-12181: --- Sure +1 > Include table name in "Cannot get comparator" exception > --- > > Key: CASSANDRA-12181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12181 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Trivial > Attachments: CASSANDRA-12181_3.0.txt > > > Having table name will help in debugging the following exception. > ERROR [MutationStage:xx] CassandraDaemon.java (line 199) Exception in thread > Thread[MutationStage:3788,5,main] > clusterName=itms8shared20 > java.lang.RuntimeException: Cannot get comparator 2 in > org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type). > > This might be due to a mismatch between the schema and the data read -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12114) Cassandra startup takes an hour because of N*N operation
[
https://issues.apache.org/jira/browse/CASSANDRA-12114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380620#comment-15380620
]
Jeff Jirsa commented on CASSANDRA-12114:
Thanks [~jkni], lgtm .
And thanks [~tvdw] for the report.
> Cassandra startup takes an hour because of N*N operation
>
>
> Key: CASSANDRA-12114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12114
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Tom van der Woerdt
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.x
>
>
> (There's a previous version of this ticket, which was very wrong about the
> actual cause. Original is quoted below)
> In java.org.cassandra.db.ColumnFamilyStore, the function scrubDataDirectories
> loops over all sstables and then for each sstable it cleans temporary files
> from its directory.
> Since there are many sstables in a directory, this ends up cleaning the same
> directory many times.
> When using leveledcompactionstrategy on a data set that is ~4TB per node, you
> can easily end up with 200k files.
> Add N and N, and we get a N*N operation (scrubDataDirectories) which ends up
> taking an hour (or more).
> (At this point I should probably point out that no, I am not sure about that.
> At all. But I do know this takes an hour and jstack blames this function)
> As promised, original ticket below :
> {quote}
> A Cassandra cluster of ours has nodes with up to 4TB of data, in a single
> table using leveled compaction having 200k files. While upgrading from 2.2.6
> to 3.0.7 we noticed that it took a while to restart a node. And with "a
> while" I mean we measured it at more than 60 minutes.
> jstack shows something interesting :
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f30db0ea400 nid=0xdb22 runnable
> [0x7f30de122000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.io.sstable.Descriptor.getTemporaryFiles(Descriptor.java:172)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:599)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> {code}
> Going by the source of File.listFiles, it puts every file in a directory into
> an array and *then* applies the filter.
> This is actually a known Java issue from 1999:
> http://bugs.java.com/view_bug.do?bug_id=4285834 -- their "solution" was to
> introduce new APIs in JRE7. I guess that makes listFiles deprecated for
> larger directories (like when using LeveledCompactionStrategy).
> tl;dr: because Cassandra uses java.io.File.listFiles, service startup can
> take an hour for larger data sets.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12114) Cassandra startup takes an hour because of N*N operation
[
https://issues.apache.org/jira/browse/CASSANDRA-12114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-12114:
---
Status: Ready to Commit (was: Patch Available)
> Cassandra startup takes an hour because of N*N operation
>
>
> Key: CASSANDRA-12114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12114
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Tom van der Woerdt
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.x
>
>
> (There's a previous version of this ticket, which was very wrong about the
> actual cause. Original is quoted below)
> In java.org.cassandra.db.ColumnFamilyStore, the function scrubDataDirectories
> loops over all sstables and then for each sstable it cleans temporary files
> from its directory.
> Since there are many sstables in a directory, this ends up cleaning the same
> directory many times.
> When using leveledcompactionstrategy on a data set that is ~4TB per node, you
> can easily end up with 200k files.
> Add N and N, and we get a N*N operation (scrubDataDirectories) which ends up
> taking an hour (or more).
> (At this point I should probably point out that no, I am not sure about that.
> At all. But I do know this takes an hour and jstack blames this function)
> As promised, original ticket below :
> {quote}
> A Cassandra cluster of ours has nodes with up to 4TB of data, in a single
> table using leveled compaction having 200k files. While upgrading from 2.2.6
> to 3.0.7 we noticed that it took a while to restart a node. And with "a
> while" I mean we measured it at more than 60 minutes.
> jstack shows something interesting :
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7f30db0ea400 nid=0xdb22 runnable
> [0x7f30de122000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.io.sstable.Descriptor.getTemporaryFiles(Descriptor.java:172)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:599)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> {code}
> Going by the source of File.listFiles, it puts every file in a directory into
> an array and *then* applies the filter.
> This is actually a known Java issue from 1999:
> http://bugs.java.com/view_bug.do?bug_id=4285834 -- their "solution" was to
> introduce new APIs in JRE7. I guess that makes listFiles deprecated for
> larger directories (like when using LeveledCompactionStrategy).
> tl;dr: because Cassandra uses java.io.File.listFiles, service startup can
> take an hour for larger data sets.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11850) cannot use cql since upgrading python to 2.7.11+
[
https://issues.apache.org/jira/browse/CASSANDRA-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380613#comment-15380613
]
Abhyuday P Singh commented on CASSANDRA-11850:
--
Thank you, it resolved the issue for now.
> cannot use cql since upgrading python to 2.7.11+
>
>
> Key: CASSANDRA-11850
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11850
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Development
>Reporter: Andrew Madison
>Assignee: Stefania
> Labels: cqlsh
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> OS: Debian GNU/Linux stretch/sid
> Kernel: 4.5.0-2-amd64 #1 SMP Debian 4.5.4-1 (2016-05-16) x86_64 GNU/Linux
> Python version: 2.7.11+ (default, May 9 2016, 15:54:33)
> [GCC 5.3.1 20160429]
> cqlsh --version: cqlsh 5.0.1
> cassandra -v: 3.5 (also occurs with 3.0.6)
> Issue:
> when running cqlsh, it returns the following error:
> cqlsh -u dbarpt_usr01
> Password: *
> Connection error: ('Unable to connect to any servers', {'odbasandbox1':
> TypeError('ref() does not take keyword arguments',)})
> I cleared PYTHONPATH:
> python -c "import json; print dir(json); print json.__version__"
> ['JSONDecoder', 'JSONEncoder', '__all__', '__author__', '__builtins__',
> '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__',
> '_default_decoder', '_default_encoder', 'decoder', 'dump', 'dumps',
> 'encoder', 'load', 'loads', 'scanner']
> 2.0.9
> Java based clients can connect to Cassandra with no issue. Just CQLSH and
> Python clients cannot.
> nodetool status also works.
> Thank you for your help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12209) Make Level compaction default
[ https://issues.apache.org/jira/browse/CASSANDRA-12209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380590#comment-15380590 ] Adam Hattrell commented on CASSANDRA-12209: --- It's pretty hard to break your cluster with STCS (it's possible but the edge cases are much harder to get to) - it's very easy with the other compaction strategies. If you could force users to properly benchmark all their workload before rolling live then, maybe, but even then I'd lean towards staying with the safe option. > Make Level compaction default > - > > Key: CASSANDRA-12209 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12209 > Project: Cassandra > Issue Type: Wish >Reporter: sankalp kohli >Priority: Minor > > Level Compaction has come a long way since it was added. I think it is time > to make it default. We can debate which version we can do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
