[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114491#comment-16114491
]
Nate McCall commented on CASSANDRA-11218:
-
[~tania.en...@quest.com] If you are seeing this consistently, turn on {{TRACE}}
logging for {{org.apache.cassandra.io.sstable.IndexSummaryRedistribution}}.
That will provide a lot of information about how much effort is being spent on
that code path.
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction, Secondary Indexes
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111658#comment-16111658 ] Tania S Engel commented on CASSANDRA-11218: --- Could the index summary redistribution be the cause of the 29 minute gap in stream time for this appliance? Typically in this test for us the Prepare completed happens within the minute. INFO [STREAM-INIT-/fd70:616e:6761:6561:ae1f:6bff:fe12:3de8:24642] 2017-07-28 *20:05*:28,929 StreamResultFuture.java:123 - [Stream #1845aa20-73d0-11e7-8027-4139c6f86357, ID#0] Received streaming plan for Bootstrap INFO [IndexSummaryManager:1] 2017-07-28 20:13:06,822 IndexSummaryRedistribution.java:75 - Redistributing index summaries INFO [STREAM-IN-/fd70:616e:6761:6561:ae1f:6bff:fe12:3de8:24642] 2017-07-28 *20:34*:10,712 StreamResultFuture.java:173 - [Stream #1845aa20-73d0-11e7-8027-4139c6f86357 ID#0] Prepare completed. Receiving 0 files(0.000KiB), sending 57 files(369.250KiB) > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Secondary Indexes >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637696#comment-15637696
]
Sam Tunnicliffe commented on CASSANDRA-11218:
-
I took a step back to re-evaulate exactly what is prioritized and when and it
turns out that adding priorities to {{CompactionTask}} itself is redundant, and
actually, in every case it's ignored. Really, it's the
{{Runnable/WrappedRunnable/Callable}} being submitted to {{CompactionExecutor}}
that needs to be prioritized and even in cases where these wrap a
{{CompactionTask}}, these have their own priority explicitly set and do not
usually inherit it from the task. That is to say, that it's the context of the
submission which determines the priority, rather than the task itself. Having
{{CompactionTask}} implement {{Prioritized}} actually makes some stuff pretty
hard to follow.
Actually, it turns out that this is somewhat academic, as most of the
priorities are not actually being observed anyway, due to the fact that
{{CompactionExecutor::submitIfRunning}} wraps everything in a
{{ListenableFutureTask}} before submission. This effectively hides any priority
that might be set so {{newTask}} always ends up creating another wrapper layer
with default priorities.
I've pushed a fix for this to my branch, and also refactored so that
{{CompactionTask}} no longer implements {{Prioritized}}, which I think
simplifies things quite a bit.
Another thing that was bugging me was that the relationship between
{{OperationType}} and {{Priority}} is somewhat muddy, as for some operations
prioritization is a meaningless concept (as indicated by the multiple special
cases). It feels like we're overloading the concepts somewhat which, for me,
again makes it much harder to grok. So I propose removing all the priority
stuff from {{OperationType}} and adding a new enum solely to represent the task
priorities (also done in my branch).
bq. I've removed the remaining uses of subtype prioritization...I've left the
logic in place for later
I think we should just rip it out if it's not being used. Like you say, if we
do come up with some good way to prioritise within types later, then we can
easily add it back, so leaving half of it hanging around in the meantime is
only going to be confusing.
The other thing that's obviously missing here is tests,
{{CompactionPriorityTest}} alone clearly isn't sufficient, seeing as it didn't
catch the fact that no prioritization was actually being done :)
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631663#comment-15631663
]
Jeff Jirsa commented on CASSANDRA-11218:
Thanks much for the review [~beobal]
Pushed
https://github.com/jeffjirsa/cassandra/commit/96288f42f36ad80bff186707f0f7234dd0a9a8d9
to address your comments (same branch
https://github.com/jeffjirsa/cassandra/commits/cassandra-11218-3.X ) .
CI @ http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.X-dtest/ and
http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.X-testall/ (started
evening of Nov 2, should be ready by Nov 3, depending on when you come back to
this).
Some notes:
{quote}
The instanceof checks and special casing to handle regular vs prioritized
runnable is also duplicated between the newTaskFor methods and the comparator.
Could we make CompactionExecutor::newTaskFor always return an
IPrioritizedCompactionFutureTask, so that when the supplied runnable/callable
is already prioritized it just uses its existing values, and when a
non-prioritized runnable/callable is received it makes it prioritized with the
default 0/0/0 values (which is functionally equivalent to what the comparator
will do anyway). Then the comparator can be further simplified by not having to
consider non-prioritized tasks.
{quote}
Seems like a good idea to me - implemented.
{quote}
On JIRA you suggested prioritizing anticompaction & index summary
redistribution joint highest, but in the patch the latter has max priority (and
is not overridable). Is that intentional?
{quote}
Index summary redistribution runs on its own
{{org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor}}, so
the priority is ignored. Added notes to reflect that.
{quote}
Validation tasks are run a dedicated executor, which doesn't use a priority
queue. This, and the fact that validation is orthogonal to compaction strategy
means that all validation tasks are equal priority & also don't block the other
tasks. So it makes the reasoning for explicity setting validation priority to
256 a little unclear.
{quote}
Now set to max value with a note that it's on its own executor.
{quote}
On a related note, CacheCleanupExecutor is also dedicated to a single task
type. This does end up using a priority queue, but only ever has vanilla
runnables submitted to it.
{quote}
Now runs with {{Cleanup}} priority (lowest). We don't really have an
OperationType for {{CacheCleanupExecutor}}, we could add one to be explicit -
perhaps this should really be same as key/row/counter cache save? I don't have
a strong opinion on what it SHOULD be.
{quote}
Some comments...
{quote}
... added.
Also - I started with the idea that we could do much, much better with
prioritizing compaction by getting very clever with subtype prioritization
(compaction within the "normal" compaction priority). Thinking about starvation
problems, and the 'right' way to generalize across compaction strategies makes
me wonder if this is a good idea or not - in this patch, I've removed the
remaining uses of subtype prioritization (where it was previously sorting tasks
by bytes on disk), in favor of the current behavior (within a given type, tasks
are ordered by timestamp). I've left the logic in place for later - if we
eventually decide there's a "right" way to prioritize tasks within a type, we
can add it back easily later. Some options that may be worth considering there
are either hotness or a per-CF flag to allow user to prioritize certain CFs
over others. In any case, I don't think it needs to be perfect - we have a huge
improvement for operations by prioritizing 2i build and deprioritizing
cleanup/scrub, so I think trying to get to perfect can wait.
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615954#comment-15615954
]
Jeff Jirsa commented on CASSANDRA-11218:
Was chatting with [~beobal] on IRC briefly about starvation, and figured it's
worth mentioning here:
{quote}
09:52 also worth thinking about - within a type/level, is size the best
comparator? or hotness?
09:52 and if size, is largest first? or smallest first?
09:52 i'm not sure there's a universal right answer
{quote}
[~krummas] and [~kohlisankalp] - do you have thoughts on ordering compaction
tasks properly within a type? Is there something that's universally more likely
to be best for the user? Hotness? Size of sstables (sorted smallest to largest,
or largest to smallest)? As written, it's doing the largest transactions first
- do either of you have strong opinions that one specific approach is right or
wrong?
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615899#comment-15615899
]
Jeff Jirsa commented on CASSANDRA-11218:
The refactor to break apart {{Priorities}} / {{Prioritized}} is much cleaner,
thanks.
{quote}
Was there any special reasoning behind making the priority fields in
PrioritizedCompactionFutureTask/PrioritizedCompactionCallable/PrioritizedCompactionWrappedRunnable
atomics rather than just final? I couldn't see how they would ever be mutated,
did I overlook something?
{quote}
I was somewhat worried about possible starvation within a type/level, where a
pathological case caused recompaction of very large files (used to happen a lot
in early DTCS), so my thought was on each compare, we could adjust
(getandincrement) the subtype priority to add an element of "how long has it
been in queue". I'm not sure if that's a real concern, so I backed out that to
reduce complexity and make it easier to reason about.
Will look at the rest this afternoon.
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615708#comment-15615708
]
Sam Tunnicliffe commented on CASSANDRA-11218:
-
[~jjirsa], sorry it's taken so long to pick up this review, I got stuck on some
unrelated stuff until just now.
The concept and general approach are straightforward & seem reasonable enough.
I had a bunch of remarks about the implementation, so I thought it would be
simpler to just push a branch
([here|https://github.com/beobal/cassandra/tree/CASSANDRA-11218) with
suggestions rather than trying to spell them out here. Let me know what you
think, the summary is:
* Refactored {{OperationType}} a bit. Aside from removing duplication, the
version in my branch normalizes the property names so that the suffix is always
the lowercase version of the type name (there are some inconsistencies in the
original version - e.g. {{KEY_CACHE_SAVE:keycachesave}} vs
{{USER_DEFINED_COMPACTION:user_defined_compaction}} vs
{{GARBAGE_COLLECT:garbage_collection}}).
* Simplified {{CompactionManager.CompactionPriorityComparator}}.
* I feel like the way priorities are carried between the various
{{IPrioritizedX}} classes is a bit messy. For example,
{{CompactionManager::submitUserDefined}} creates an anonymous instance of
{{PriotitizedCompactionWrappedRunnable}} that has a timestamp set in it's
constructor. That's then submitted to the compaction executor, where
{{newTaskFor}} wraps it in an {{IPrioritizedCompactionFutureTask}} using the
runnable's type and subtype priority, but taking a new timestamp, dropping the
one on the runnable. Ultimately, I ended up encapsulating the 3 priority values
and just using that {{Priorities}} class everywhere (I also renamed
{{IPrioritizedCompactionComparable}} to {{Prioritized}}).
** Was there any special reasoning behind making the priority fields in
{{PrioritizedCompactionFutureTask}}/{{PrioritizedCompactionCallable}}/{{PrioritizedCompactionWrappedRunnable}}
atomics rather than just final? I couldn't see how they would ever be mutated,
did I overlook something?
One thought that I didn't have chance to explore in that branch:
* The {{instanceof}} checks and special casing to handle regular vs prioritized
runnable is also duplicated between the {{newTaskFor}} methods and the
comparator. Could we make {{CompactionExecutor::newTaskFor}} always return an
{{IPrioritizedCompactionFutureTask}}, so that when the supplied
runnable/callable is already prioritized it just uses its existing values, and
when a non-prioritized runnable/callable is received it makes it prioritized
with the default 0/0/0 values (which is functionally equivalent to what the
comparator will do anyway). Then the comparator can be further simplified by
not having to consider non-prioritized tasks.
And a few more minor points:
* On JIRA you suggested prioritizing anticompaction & index summary
redistribution joint highest, but in the patch the latter has max priority (and
is not overridable). Is that intentional?
* Validation tasks are run a dedicated executor, which doesn't use a priority
queue. This, and the fact that validation is orthogonal to compaction strategy
means that all validation tasks are equal priority & also don't block the other
tasks. So it makes the reasoning for explicity setting validation priority to
256 a little unclear.
* On a related note, {{CacheCleanupExecutor}} is also dedicated to a single
task type. This *does* end up using a priority queue, but only ever has vanilla
runnables submitted to it.
Finally:
Some comments in {{IPrioritizedCompactionComparable/Prioritized}} describing
its purpose & maybe the comparison algorithm (e.g. what's described in the
comments here, above) could be useful.
Also some comments in {{OperationType}} on the fact that priority relates to
task scheduling in compaction manager, and why some types have un-overridable
priority of {{MAX_VALUE}} might be handy.
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1530#comment-1530 ] Jeff Jirsa commented on CASSANDRA-11218: [~beobal] - skip review on the 3.0 branch, probably a big enough change we shouldn't push it to 3.0 at this point. However, here's 3.X: | [3.X|https://github.com/jeffjirsa/cassandra/tree/cassandra-11218-3.X] | [utest|http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.X-testall/] | [dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.X-dtest/] | > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526789#comment-15526789 ] Joshua McKenzie commented on CASSANDRA-11218: - [~beobal] to review. > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[
https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511926#comment-15511926
]
Jeff Jirsa commented on CASSANDRA-11218:
Branch here:
https://github.com/jeffjirsa/cassandra/commit/50dc4ed58a33ac76532eb9eaaa40c64d65bf3489
Test here:
http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.0-testall/lastBuild/
Dtest: http://cassci.datastax.com/job/jeffjirsa-cassandra-11218-3.0-dtest/
(dtest is aborting, which I'm investigating).
I thought we had a problem with {{LongLeveledCompactionStrategyTest}} in
testall build #3, then I found CASSANDRA-12202 and CASSANDRA-11657 , and I'm
less concerned with it (long story short, it's flakey in 3.0, but it seems
stable if I backport the 11657 patch).
[~krummas] , I suspect you're best qualified to review - mind if I add you as
reviewer? Have strong opinion on which version should get this (3.0 vs 3.10?)
> Prioritize Secondary Index rebuild
> --
>
> Key: CASSANDRA-11218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11218
> Project: Cassandra
> Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
>
> We have seen that secondary index rebuild get stuck behind other compaction
> during a bootstrap and other operations. This causes things to not finish. We
> should prioritize index rebuild via a separate thread pool or using a
> priority queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508506#comment-15508506 ] Jeff Jirsa commented on CASSANDRA-11218: Branch here for 3.0: https://github.com/jeffjirsa/cassandra/commit/50dc4ed58a33ac76532eb9eaaa40c64d65bf3489 testall/dtests queued, will link when they complete. > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502593#comment-15502593 ] Marcus Eriksson commented on CASSANDRA-11218: - sounds good to me > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502288#comment-15502288 ] Jeff Jirsa commented on CASSANDRA-11218: I have a version of this patch I'll be submitting very soon, but while I wait for internal approvals, I'd like to describe the implementation so that those of you who care about this can provide feedback conceptually before I submit a patch for review. I'm implementing this as a priority queue that uses a custom comparator implemented with three tiers: * Operation type priority (to allow certain types - like index rebuild - to run at higher priorities, and others - scrub / cleanup / verify - to run at much lower priorities). This is defined as an int field in the enum in the OperationType, and can be overridden via system property. Lot of opportunity for bike shedding here in picking exact priorities - I've chosen (highest priority to lowest): ** Anticompaction ** Index Build / View Build ** Key Cache Save / Row Cache Save / Counter Cache Save ** User Defined Compaction ** Compaction (including maximal/major compaction) ** Tombstone Compaction ** Scrub / Cleanup / Upgrade SSTables ** Index Summary Redistribution ** Verify * Sub type priority (to allow compaction tasks within a type to have preference - to enable behavior like CASSANDRA-6288 ). This is defined as a long, and set by the compaction strategies, and by default, I'm setting this as the bytes on disk of the source sstables - larger transactions (at the time the task was created) preferred over smaller transactions. * Timestamp priority, where tasks with the same type/subtype values are served FIFO. The implementation here was pretty straight forward - we create a new interface to expose the three priority values, and then extend AbstractCompactionTask and de-anonymize the handful of anonymous runnables/wrapped runnables/callables to implement that interface so they can be sorted in the PriorityBlockingQueue. There may an opportunity to try to get clever to protect against starvation in under-resourced systems, such as increasing type priority over time as tasks age, but I'm leaving that as a potential optimization for the future - I'm not sure it's really needed, it makes reasoning about compaction harder, but maybe there exists a use case where it's necessary. Expecting to submit the patch early this week - if either of you (Sankalp / Marcus) finds this approach conflicts with your expectations, or if you want to volunteer to review, let me know. > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15168316#comment-15168316 ] Jeff Jirsa commented on CASSANDRA-11218: Somewhat tangential to this ticket, but if you add priority queue / compaction priorities, could also de-prioritize cleanup so sstable counts dont skyrocket during parallel cleanup? > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167884#comment-15167884 ] Marcus Eriksson commented on CASSANDRA-11218: - I agree, it feels like the cleanest approach - waiting for the currently executing task is probably fine, but not waiting for all the queued tasks AFAIK we don't use this approach anywhere else > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167878#comment-15167878 ] sankalp kohli commented on CASSANDRA-11218: --- [~krummas] I think using PriorityQueue approach is better than adding a separate thread pool. What do you think? Are we using something like this any where else in the code to prioritize runnables? > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166991#comment-15166991 ] Marcus Eriksson commented on CASSANDRA-11218: - yep, makes sense > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159403#comment-15159403 ] sankalp kohli commented on CASSANDRA-11218: --- cc [~krummas] I think we should also prioritize user defined compactions over others. What do you think? > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
