[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109177#comment-15109177 ] Jeff Ferland commented on CASSANDRA-10979: -- Applied the patch to a node that had just come up from streaming and was 700+ tables in L0. After restart, observed that as L0 was shifted into L1, L0 continued to compact new tables to prevent the extreme growth of tables. Thank you. Again still requesting a merge into 2.1 since streaming and repair with LCS are practically broken for us without this patch. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109648#comment-15109648 ] Carl Yeksigian commented on CASSANDRA-10979: I've rebased and pushed up new 2.2-trunk branches. ||2.2||3.0||3.3||trunk|| |[branch|https://github.com/carlyeks/cassandra/tree/ticket/10979/2.2]|[branch|https://github.com/carlyeks/cassandra/tree/ticket/10979/3.0]|[branch|https://github.com/carlyeks/cassandra/tree/ticket/10979/3.3]|[branch|https://github.com/carlyeks/cassandra/tree/ticket/10979/trunk]| |[utest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-2.2-testall/]|[utest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-3.0-testall/]|[utest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-3.3-testall/]|[utest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-trunk-testall/]| |[dtest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-3.3-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-10979-trunk-dtest/]| > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 2.2.x, 3.0.x, 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098279#comment-15098279 ] Jeff Ferland commented on CASSANDRA-10979: -- [~sebastian.este...@datastax.com]: Can you cut a build of DSE 4.8.4 with this patch added? I'll try my best to test within a week. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095881#comment-15095881 ] Marcus Eriksson commented on CASSANDRA-10979: - [~autocracy] have you confirmed that the patch fixes your issue? I would assume that any slowdown after incremental repair is due to https://issues.apache.org/jira/browse/CASSANDRA-10831 > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096481#comment-15096481 ] Carl Yeksigian commented on CASSANDRA-10979: I'm fine with this going in 2.2 as it's not an intrusive fix, but I also think this is more of an improvement than a bug fix. While this will reduce the number of sstables on a read when L0 is backed up, the problem remains that L0 cannot keep up with the amount of incoming data. Once we finish the L0->L1 compaction, we will be running the L0 STCS since we will have a level that is oversized which will also cause us to check L0 again. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093818#comment-15093818 ] Marcus Eriksson commented on CASSANDRA-10979: - This LGTM, could you push a branch so we get the test runs in? And, should we really target 2.1 for this? > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 2.1.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095112#comment-15095112 ] Jeremiah Jordan commented on CASSANDRA-10979: - Agree with Jeff here. This seems like a bug in the intended behavior that we should at least fix in 2.2+, since it's not a critical bug fix to go in 2.1. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094350#comment-15094350 ] Jeff Ferland commented on CASSANDRA-10979: -- The problem I've got and why I considered it a bug is that the behavior as it is right now doesn't appear to be as it was intended, and particularly negative for me is that it destroys performance of my cluster by an order of magnitude with LCS tables during a repair as potentially thousands of tables build up in L0. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088139#comment-15088139 ] Jeff Ferland commented on CASSANDRA-10979: -- Debug logging using the current code: https://gist.github.com/autocracy/346afa253175af475770 > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland > Labels: compaction, leveled > Fix For: 2.1.x > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[
https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088274#comment-15088274
]
Sebastian Estevez commented on CASSANDRA-10979:
---
[~krummas] I mentioned this briefly this morning over chat but now we have more
information specifically we observe a large L0 to L1 compaction that blocks L0
STCS compactions, probably because the remaining L0 sstables overlap
compactingL0. You can see in the log above that {quote}L0 is too far behind,
performing size-tiering there first{quote} does not appear. What do you think?
> LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
> -
>
> Key: CASSANDRA-10979
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10979
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: 2.1.11 / 4.8.3 DSE.
>Reporter: Jeff Ferland
> Labels: compaction, leveled
> Fix For: 2.1.x
>
>
> Reading code from
> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
> and comparing with behavior shown in
> https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens:
> Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to
> incremental repair).
> Relevant code from here is
> if (Sets.intersection(l1overlapping, compacting).size() > 0)
> return Collections.emptyList();
> Since there will be overlap between what is compacting and L1 (in my case,
> pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000
> smaller tables in L0 while awaiting the transition from L0 to L1 and destroy
> my performance.
> Requested outcome is to continue to perform SCTS on non-compacting L0 tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
