[
https://issues.apache.org/jira/browse/CASSANDRA-12114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380531#comment-15380531
]
Joel Knighton commented on CASSANDRA-12114:
-------------------------------------------
Thanks for the patch. I made a tiny whitespace fix and removed the absolute
path conversion of the already canonicalized descriptor directory. You are
correct that the failures in your utests/dtests runs are unrelated. CI for my
slight tweaks here - testall runs were initially noisy due to some
environmental problems, but I've produced clean relative to upstream runs once
those were fixed.
||branch||testall||dtest||
|[CASSANDRA-12114-3.0|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.0-dtest]|
|[CASSANDRA-12114-3.9|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-3.9]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.9-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.9-dtest]|
|[CASSANDRA-12114-trunk|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-trunk-dtest]|
The 3.0 patch merges forward cleanly through 3.9 and trunk. Although this issue
is tagged as an improvement and thus would typically not go to 3.0.x or 3.x,
since the performance impact can be so severe in some cases, I think we should
merge it into these bugfix branches.
If you're fine with the small tweaks I made [~jjirsa], feel free to set this to
'Ready to Commit'.
> Cassandra startup takes an hour because of N*N operation
> --------------------------------------------------------
>
> Key: CASSANDRA-12114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12114
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Tom van der Woerdt
> Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.x
>
>
> (There's a previous version of this ticket, which was very wrong about the
> actual cause. Original is quoted below)
> In java.org.cassandra.db.ColumnFamilyStore, the function scrubDataDirectories
> loops over all sstables and then for each sstable it cleans temporary files
> from its directory.
> Since there are many sstables in a directory, this ends up cleaning the same
> directory many times.
> When using leveledcompactionstrategy on a data set that is ~4TB per node, you
> can easily end up with 200k files.
> Add N and N, and we get a N*N operation (scrubDataDirectories) which ends up
> taking an hour (or more).
> (At this point I should probably point out that no, I am not sure about that.
> At all. But I do know this takes an hour and jstack blames this function)
> As promised, original ticket below :
> {quote}
> A Cassandra cluster of ours has nodes with up to 4TB of data, in a single
> table using leveled compaction having 200k files. While upgrading from 2.2.6
> to 3.0.7 we noticed that it took a while to restart a node. And with "a
> while" I mean we measured it at more than 60 minutes.
> jstack shows something interesting :
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x00007f30db0ea400 nid=0xdb22 runnable
> [0x00007f30de122000]
> java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.io.sstable.Descriptor.getTemporaryFiles(Descriptor.java:172)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:599)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> {code}
> Going by the source of File.listFiles, it puts every file in a directory into
> an array and *then* applies the filter.
> This is actually a known Java issue from 1999:
> http://bugs.java.com/view_bug.do?bug_id=4285834 -- their "solution" was to
> introduce new APIs in JRE7. I guess that makes listFiles deprecated for
> larger directories (like when using LeveledCompactionStrategy).
> tl;dr: because Cassandra uses java.io.File.listFiles, service startup can
> take an hour for larger data sets.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
