[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020171#comment-13020171 ] Chris Goffinet edited comment on CASSANDRA-1902 at 4/15/11 3:19 AM: CASSANDRA-1470 introduced a way to help not pollute page cache. It also didn't solve it 100% We realized that during compaction, DONTNEED is used when reading pages from existing SSTables. While compaction runs this will effectively make all hot pages in those files, which are still being served be marked to be removed from cache. So we certainly helped not pollute cache by taking in cold data, we also effectively removed hot data as well while compaction runs. We need to modify this patch so that while reading pages on existing SSTables we be sure to not remove those hot pages during compaction. I am happy to open another ticket if that makes more sense. But I figured we could just go ahead and make this ticket support that functionality too. was (Author: lenn0x): CASSANDRA-1470 introduced a way to help not pollute page cache. It also didn't solve it 100% We realized that during compaction, DONTNEED is used when reading pages from existing SSTables. While compaction runs this will effectively make all hot pages in those files, which are still being served be marked to be removed from cache. So we certainly helped not pollute cache, we also effectively removed hot data as well while compaction runs. We need to modify this patch so that while reading pages on existing SSTables we be sure to not remove those hot pages during compaction. I am happy to open another ticket if that makes more sense. But I figured we could just go ahead and make this ticket support that functionality too. > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 1.0 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, > CASSANDRA-1902-v9-trunk-rebased.patch, > CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, > CASSANDRA-1902-v9.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016551#comment-13016551
]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/6/11 9:38 PM:
removed @finalize from MappedFileDataInput which was dropping page cache. Now I
see the following results with migration enabled:
options: -S 1024 -c 10 -n 10 (server 4 GB RAM, Quad-Core AMD Opteron(tm)
Processor 2374 HE)
{noformat}
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
--- after compaction --
13 12 14 13
{noformat}
was (Author: xedin):
removed @finalize from MappedFileDataInput which was dropping page cache.
Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 10 (server 4 GB RAM, Quad-Core AMD Opteron(tm)
Processor 2374 HE)
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
--- after compaction --
13 12 14 13
> Migrate cached pages during compaction
> ---
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.7.1
>Reporter: T Jake Luciani
>Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments:
> 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt,
> 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt,
> 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt,
> CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch,
> CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a
> pre-compacted CF during the compaction process. This is now important since
> CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached
> in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from
> active CFs. This can be done my monitoring which Old SSTables are in the page
> cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here:
> http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016551#comment-13016551 ] Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/6/11 9:37 PM: removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled: options: -S 1024 -c 10 -n 10 (server 4 GB RAM, Quad-Core AMD Opteron(tm) Processor 2374 HE) 188 (write) 16 (read) 14 (read) 191 (write) --- reads during compaction --- 21 16 16 27 41 22 16 15 14 15 16 18 14 13 --- after compaction -- 13 12 14 13 was (Author: xedin): removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled: options: -S 1024 -c 10 -n 10 188 (write) 16 (read) 14 (read) 191 (write) --- reads during compaction --- 21 16 16 27 41 22 16 15 14 15 16 18 14 13 --- after compaction 13 12 14 13 > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015973#comment-13015973 ] T Jake Luciani edited comment on CASSANDRA-1902 at 4/5/11 3:58 PM: --- Minor issue: Ran stress tool -o read against a empty db with migration enabled and got the following: WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11. WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11. was (Author: tjake): Ran stress tool -o read against a empty db with migration enabled and got the following: WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11. WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11. > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015815#comment-13015815 ] Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/5/11 9:42 AM: uses SSTableReader.dfile by mincore instead of re-mmaping regions all the time in the mmaped mode, if mode is standard uses CLibrary.getCachedPages() method was (Author: xedin): uses SSTable.dfile by mincore instead of re-mmaping regions all the time in the mmaped mode, if mode is standard uses CLibrary.getCachedPages() method > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014001#comment-13014001 ] Pavel Yaskevich edited comment on CASSANDRA-1902 at 3/31/11 3:13 PM: - What is the difference with the current code? was (Author: xedin): What is the different with the current code? > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002637#comment-13002637 ] Pavel Yaskevich edited comment on CASSANDRA-1902 at 3/4/11 3:00 PM: Rebased patch does not include org.apache.cassandra.utils.PageCacheMetrics and PageCacheInformer classes. was (Author: xedin): Rebased patch does not include org.apache.cassandra.utils.PageCacheMetrics class. > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 0.7.4 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-formatted.txt, 1902-per-column-migration-rebase.txt, > 1902-per-column-migration.txt > > Original Estimate: 32h > Time Spent: 24h > Remaining Estimate: 8h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. > First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that > uses the posix mincore() function to detect the offsets of pages for this > file currently in page cache. > Then add getActiveKeys() which uses underlying pagesInPageCache() to get the > keys actually in the page cache. > use getActiveKeys() to detect which SSTables being compacted are in the os > cache and make sure the subsequent pages in the new compacted SSTable are > kept in the page cache for these keys. This will minimize the impact of > compacting a "hot" SSTable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996199#comment-12996199 ] Chris Goffinet edited comment on CASSANDRA-1902 at 2/18/11 1:52 AM: Couldn't we build a list when reading/writing of the SSTables the first time, so that after new SSTable write is finished, we just mmap through the page cache, and we already know the new locations on the first pass? was (Author: lenn0x): Couldn't we build a list when reading/writing of the mappings, so that after write is finished, we just mmap through the page cache, and we already know the new locations on the first pass? > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 0.7.3 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt > > Original Estimate: 32h > Time Spent: 24h > Remaining Estimate: 8h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. > First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that > uses the posix mincore() function to detect the offsets of pages for this > file currently in page cache. > Then add getActiveKeys() which uses underlying pagesInPageCache() to get the > keys actually in the page cache. > use getActiveKeys() to detect which SSTables being compacted are in the os > cache and make sure the subsequent pages in the new compacted SSTable are > kept in the page cache for these keys. This will minimize the impact of > compacting a "hot" SSTable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978450#action_12978450 ] T Jake Luciani edited comment on CASSANDRA-1902 at 1/6/11 1:46 PM: --- Right, *not* DONTNEEDing the pages needed should prob be fine just thought WILLNEED might ensure these pages are kept over the presumably active older sstables. So you are recommending a binary flag like "compaction.hotness.migration.thresh = .5"? I was thinking the same for simplicity sake. was (Author: tjake): Right, *not* DONTNEEDing the pages needed should prob be fine just though will need might ensure these pages are kept over the presumably active older sstables. So you are recommending a binary flag like "compaction.hotness.migration.thresh = .5"? I was thinking the same for simplicity sake. > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 0.7.1 > > Original Estimate: 32h > Remaining Estimate: 32h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. > First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that > uses the posix mincore() function to detect the offsets of pages for this > file currently in page cache. > Then add getActiveKeys() which uses underlying pagesInPageCache() to get the > keys actually in the page cache. > use getActiveKeys() to detect which SSTables being compacted are in the os > cache and make sure the subsequent pages in the new compacted SSTable are > kept in the page cache for these keys. This will minimize the impact of > compacting a "hot" SSTable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
