[jira] [Commented] (CASSANDRA-15306) Investigate why we are allocating 8MiB chunks and reaching the maximum BufferPool size
[
https://issues.apache.org/jira/browse/CASSANDRA-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062227#comment-17062227
]
Joey Lynch commented on CASSANDRA-15306:
It was about 150 WPS against a LCS table with 4KiB single column partitions. If
this is proving hard to reproduce I'm cool with closing as "cannot reproduce"
and if we see it again I'll re-open and do more digging.
> Investigate why we are allocating 8MiB chunks and reaching the maximum
> BufferPool size
> --
>
> Key: CASSANDRA-15306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15306
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging, Test/benchmark
>Reporter: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
>
> While throwing some light traffic at {{4.0-alpha1}} I saw a lot of the
> following in the logs
> {noformat}
> INFO [CompactionExecutor:8] 2019-09-06 11:40:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:8] 2019-09-06 11:55:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:15] 2019-09-06 12:10:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:18] 2019-09-06 12:25:31,421 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> {noformat}
> This was with about 150 WPS against a LCS table containing 4kib partitions.
> It seemed that compaction proceeded just fine but I don't remember seeing
> this in previous testing runs and I'd like to make sure it's not a bug
> (otherwise we may want to reduce the logging).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15306) Investigate why we are allocating 8MiB chunks and reaching the maximum BufferPool size
[
https://issues.apache.org/jira/browse/CASSANDRA-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061961#comment-17061961
]
David Capwell commented on CASSANDRA-15306:
---
[~jolynch] can you define "light traffic"? if you are doing something like 10
qps and you see this then something is very off. If you are doing 1000 qps per
node then this may be expected?
All my tests filter for WARN/ERROR so don't have history to say if I see this
with my tests.
> Investigate why we are allocating 8MiB chunks and reaching the maximum
> BufferPool size
> --
>
> Key: CASSANDRA-15306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15306
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging, Test/benchmark
>Reporter: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
>
> While throwing some light traffic at {{4.0-alpha1}} I saw a lot of the
> following in the logs
> {noformat}
> INFO [CompactionExecutor:8] 2019-09-06 11:40:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:8] 2019-09-06 11:55:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:15] 2019-09-06 12:10:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:18] 2019-09-06 12:25:31,421 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> {noformat}
> This was with about 150 WPS against a LCS table containing 4kib partitions.
> It seemed that compaction proceeded just fine but I don't remember seeing
> this in previous testing runs and I'd like to make sure it's not a bug
> (otherwise we may want to reduce the logging).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15306) Investigate why we are allocating 8MiB chunks and reaching the maximum BufferPool size
[
https://issues.apache.org/jira/browse/CASSANDRA-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061462#comment-17061462
]
ZhaoYang commented on CASSANDRA-15306:
--
8MB is the size of macro chunk in global buffer pool... Should we close this
ticket?
> Investigate why we are allocating 8MiB chunks and reaching the maximum
> BufferPool size
> --
>
> Key: CASSANDRA-15306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15306
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging, Test/benchmark
>Reporter: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
>
> While throwing some light traffic at {{4.0-alpha1}} I saw a lot of the
> following in the logs
> {noformat}
> INFO [CompactionExecutor:8] 2019-09-06 11:40:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:8] 2019-09-06 11:55:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:15] 2019-09-06 12:10:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:18] 2019-09-06 12:25:31,421 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> {noformat}
> This was with about 150 WPS against a LCS table containing 4kib partitions.
> It seemed that compaction proceeded just fine but I don't remember seeing
> this in previous testing runs and I'd like to make sure it's not a bug
> (otherwise we may want to reduce the logging).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15306) Investigate why we are allocating 8MiB chunks and reaching the maximum BufferPool size
[
https://issues.apache.org/jira/browse/CASSANDRA-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050724#comment-17050724
]
David Capwell commented on CASSANDRA-15306:
---
Given the fact this is compaction this wold most likely be
org.apache.cassandra.cache.ChunkCache I think.
org.apache.cassandra.utils.memory.BufferPool.LocalPool#get will attempt to grow
until this limit is reached and then will avoid using the pool and rely on GC
to cleanup buffers, currently buffers are heap but CASSANDRA-15358 wants to
switch to direct (needed by netty).
To replicate you can set file_cache_size_in_mb to a very small value, say 0 or
1 (never tried with such a value so may see other issues); this pool is shared
between internode networking and check cache
> Investigate why we are allocating 8MiB chunks and reaching the maximum
> BufferPool size
> --
>
> Key: CASSANDRA-15306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15306
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging, Test/benchmark
>Reporter: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
>
> While throwing some light traffic at {{4.0-alpha1}} I saw a lot of the
> following in the logs
> {noformat}
> INFO [CompactionExecutor:8] 2019-09-06 11:40:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:8] 2019-09-06 11:55:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:15] 2019-09-06 12:10:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:18] 2019-09-06 12:25:31,421 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> {noformat}
> This was with about 150 WPS against a LCS table containing 4kib partitions.
> It seemed that compaction proceeded just fine but I don't remember seeing
> this in previous testing runs and I'd like to make sure it's not a bug
> (otherwise we may want to reduce the logging).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15306) Investigate why we are allocating 8MiB chunks and reaching the maximum BufferPool size
[
https://issues.apache.org/jira/browse/CASSANDRA-15306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031984#comment-17031984
]
Brandon Williams commented on CASSANDRA-15306:
--
I was not able to reproduce this. If you still see it on trunk, can you give
more details?
> Investigate why we are allocating 8MiB chunks and reaching the maximum
> BufferPool size
> --
>
> Key: CASSANDRA-15306
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15306
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Logging, Test/benchmark
>Reporter: Joey Lynch
>Priority: Normal
> Fix For: 4.0-beta
>
>
> While throwing some light traffic at {{4.0-alpha1}} I saw a lot of the
> following in the logs
> {noformat}
> INFO [CompactionExecutor:8] 2019-09-06 11:40:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:8] 2019-09-06 11:55:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:15] 2019-09-06 12:10:31,419 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> INFO [CompactionExecutor:18] 2019-09-06 12:25:31,421 NoSpamLogger.java:91 -
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of 8.000MiB
> {noformat}
> This was with about 150 WPS against a LCS table containing 4kib partitions.
> It seemed that compaction proceeded just fine but I don't remember seeing
> this in previous testing runs and I'd like to make sure it's not a bug
> (otherwise we may want to reduce the logging).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
