[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15388: Source Control Link: https://github.com/apache/cassandra/commit/c9e40277f0389a773b21d4310a85a8c761932d61 Resolution: Fixed Status: Resolved (was: Ready to Commit) thanks, committed to trunk as [c9e40277f0389a773b21d4310a85a8c761932d61|https://github.com/apache/cassandra/commit/c9e40277f0389a773b21d4310a85a8c761932d61] > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15388: Status: Ready to Commit (was: Changes Suggested) > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15388: -- Status: Changes Suggested (was: Review In Progress) > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15388: -- Reviewers: Benedict Elliott Smith, David Capwell, David Capwell (was: Benedict Elliott Smith, David Capwell) Benedict Elliott Smith, David Capwell, David Capwell (was: Benedict Elliott Smith) Status: Review In Progress (was: Patch Available) Personally would love if test-memory target could go away in favor of testclasslist so it matches the rest of CI. There is only one test right now so not a big issue, but something that should change if "enough" tests get added. If you don't want to add this to circle ci in this, can you file a follow up JIRA for Circle CI? > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15388: Reviewers: Benedict Elliott Smith > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15388: Test and Documentation Plan: circle Status: Patch Available (was: Open) [4.0|https://github.com/bdeggleston/cassandra/tree/15388] > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15388: Change Category: Performance Complexity: Normal Fix Version/s: 4.0 Status: Open (was: Triage Needed) > Add compaction allocation measurement test to support compaction gc > optimization. > -- > > Key: CASSANDRA-15388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15388 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > This adds a test that is able to quickly and accurately measure the effect of > potential gc optimizations against a wide range of (synthetic) compaction > workloads. This test accurately measures allocation rates from 16 workloads > in less that 2 minutes. > This test uses google’s {{java-allocation-instrumenter}} agent to measure the > workloads. Measurements using this agent are very accurate and pretty > repeatable from run to run, with most variance being negligible (1-2 bytes > per partition), although workloads with larger but fewer partitions vary a > bit more (still less that 0.03%). > The thinking behind this patch is that with compaction, we’re generally > interested in the memory allocated per partition, since garbage scales more > or less linearly with the number of partitions compacted. So measuring > allocation from a small number of partitions that otherwise represent real > world use cases is a good enough approximation. > In addition to helping with compaction optimizations, this test could be used > as a template for future optimization work. This pattern could also be used > to set allocation limits on workloads/operations and fail CI if the > allocation behavior changes past some threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org