[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5661: -- Fix Version/s: (was: 1.2.10) > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661-global-multiway-cache.patch, > CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: CASSANDRA-5661-global-multiway-cache.patch this is updated patch which actually includes onRemoval callback. > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661-global-multiway-cache.patch, > CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: (was: CASSANDRA-5661-global-multiway-cache.patch) > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661-global-multiway-cache.patch, > CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: CASSANDRA-5661-global-multiway-cache.patch attaching patch with supports the latest version of multiway pool (and latest trunk) and adds (C)RAR deallocation on object removal. Sorry I didn't run the test sooner, I was quiet busy with other things... So I see that with this version 95-99.9 have degraded comparing to previous by ~3 ms but they don't shake as they used too - it's around ~5 ms now (I'm testing with expireAfterAccess). > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661-global-multiway-cache.patch, > CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: (was: CASSANDRA-5661-multiway-per-sstable.patch) > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: (was: CASSANDRA-5661-v2-global-multiway-per-sstable.patch) > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5661: -- Fix Version/s: (was: 2.0) 2.0.1 > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0.1 > > Attachments: CASSANDRA-5661-multiway-per-sstable.patch, > CASSANDRA-5661.patch, CASSANDRA-5661-v2-global-multiway-per-sstable.patch, > DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: (was: CASSANDRA-5661-v2-global-multiway-per-sstable.patch) > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0 > > Attachments: CASSANDRA-5661-multiway-per-sstable.patch, > CASSANDRA-5661.patch, CASSANDRA-5661-v2-global-multiway-per-sstable.patch, > DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: CASSANDRA-5661-v2-global-multiway-per-sstable.patch v2 updated with a small thing - added recordStats() to the builder so we have statistics information available on demand. Exception listed above still accours. > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0 > > Attachments: CASSANDRA-5661-multiway-per-sstable.patch, > CASSANDRA-5661.patch, CASSANDRA-5661-v2-global-multiway-per-sstable.patch, > DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[
https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Yaskevich updated CASSANDRA-5661:
---
Attachment: CASSANDRA-5661-v2-global-multiway-per-sstable.patch
bq. Can you elaborate?
Please that a look to v2 patch PoolingSegmentFile.getSegment() method, I have
added comment about that.
v2 patch makes cache global and adds per borrow loading of missing items. The
problem I encountered is that system is unable to start because of the
following error:
I'm not sure if I'm doing something wrong or it's a bug in multiway pool
{noformat}
INFO 22:19:07,254 Opening
/var/lib/cassandra/data/system/local/system-local-ja-21 (520 bytes)
ERROR 22:19:07,298 Exception encountered during startup
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.NullPointerException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4793)
at
com.github.benmanes.multiway.TransferPool.newResourceHandle(TransferPool.java:206)
at
com.github.benmanes.multiway.TransferPool.tryToGetResourceHandle(TransferPool.java:186)
at
com.github.benmanes.multiway.TransferPool.getResourceHandle(TransferPool.java:167)
at
com.github.benmanes.multiway.TransferPool.borrow(TransferPool.java:152)
at
com.github.benmanes.multiway.TransferPool.borrow(TransferPool.java:143)
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:64)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1040)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:96)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:109)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:62)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:87)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:124)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1458)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1284)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
at
org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:227)
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:245)
at
org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:56)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:154)
at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:456)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:237)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:453)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:496)
Caused by: java.lang.NullPointerException
at
com.github.benmanes.multiway.TransferPool$2.call(TransferPool.java:209)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4796)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
... 28 more
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.NullPointerException
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4793)
at
com.github.benmanes.multiway.TransferPool.newResourceHandle(TransferPool.java:206)
at
com.github.benmanes.multiway.TransferPool.tryToGetResourceHandle(TransferPool.java:186)
at
com.github.benmanes.multiway.TransferPool.getResourceHandle(TransferPool.java:167)
at
com.github.benmanes.multiway.TransferPool.borrow
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: CASSANDRA-5661-multiway-per-sstable.patch > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0 > > Attachments: CASSANDRA-5661-multiway-per-sstable.patch, > CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5661: -- Fix Version/s: (was: 1.2.7) 2.0 > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 2.0 > > Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-5661: --- Attachment: CASSANDRA-5661.patch Attached patch adds FileCacheService (+ metrics) which can expire instances after they are not accessed for time period and has a global memory usage limit (set to 512MB). > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 1.2.7 > > Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremiah Jordan updated CASSANDRA-5661: --- Attachment: DominatorTree.png Histogram.png > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 1.2.7 > > Attachments: DominatorTree.png, Histogram.png > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5661) Discard pooled readers for cold data
[ https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cathy Daw updated CASSANDRA-5661: - Tester: dmeyer > Discard pooled readers for cold data > > > Key: CASSANDRA-5661 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5661 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.1 >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 1.2.7 > > > Reader pooling was introduced in CASSANDRA-4942 but pooled > RandomAccessReaders are never cleaned up until the SSTableReader is closed. > So memory use is "the worst case simultaneous RAR we had open for this file, > forever." > We should introduce a global limit on how much memory to use for RAR, and > evict old ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
