Re: Cache full text into memory
I doubt about it. Caching system is a key value store. You have to use some compression library to compress and decompress your data. Caching system helps to retrieve fast. Anyways please take a look of each of the caching system features. Regards Aditya www.findbestopensource.com On Wed, Jul 14, 2010 at 3:06 PM, Li Li wrote: > Thank you. I don't know which cache system to use. In my application, > the cache system must support compression algorithm which has high > compression ratio and fast decompression speed(because each time it > get from cache, it must decompress). > > 2010/7/14 findbestopensource : > > I have just provided you two options. Since you already store as part of > the > > index, You could try external caching. Try using ehcache / Membase > > http://www.findbestopensource.com/tagged/distributed-caching . The > caching > > system will do LRU and is much more efficient. > > > > On Wed, Jul 14, 2010 at 12:39 PM, Li Li wrote: > > > >> I have already store it in lucene index. But it is in disk and When a > >> query come, it must seek the disk to get it. I am not familiar with > >> lucene cache. I just want to fully use my memory that load 10GB of it > >> in memory and a LRU stragety when cache full. To load more into > >> memory, I want to compress it "in memory". I don't care much about > >> disk space so whether or not it's compressed in lucene . > >> > >> 2010/7/14 findbestopensource : > >> > You have two options > >> > 1. Store the compressed text as part of stored field in Solr. > >> > 2. Using external caching. > >> > http://www.findbestopensource.com/tagged/distributed-caching > >> >You could use ehcache / Memcache / Membase. > >> > > >> > The problem with external caching is you need to synchronize the > >> deletions > >> > and modification. Fetching the stored field from Solr is also faster. > >> > > >> > Regards > >> > Aditya > >> > www.findbestopensource.com > >> > > >> > > >> > On Wed, Jul 14, 2010 at 12:08 PM, Li Li wrote: > >> > > >> >> I want to cache full text into memory to improve performance. > >> >> Full text is only used to highlight in my application(But it's very > >> >> time consuming, My avg query time is about 250ms, I guess it will > cost > >> >> about 50ms if I just get top 10 full text. Things get worse when get > >> >> more full text because in disk, it scatters erverywhere for a > query.). > >> >> My full text per machine is about 200GB. The memory available for > >> >> store full text is about 10GB. So I want to compress it in memory. > >> >> Suppose compression ratio is 1:5, then I can load 1/4 full text in > >> >> memory. I need a Cache component for it. Has anyone faced the problem > >> >> before? I need some advice. Is it possbile using external tools such > >> >> as MemCached? Thank you. > >> >> > >> > > >> > > >
Re: Cache full text into memory
Thank you. I don't know which cache system to use. In my application, the cache system must support compression algorithm which has high compression ratio and fast decompression speed(because each time it get from cache, it must decompress). 2010/7/14 findbestopensource : > I have just provided you two options. Since you already store as part of the > index, You could try external caching. Try using ehcache / Membase > http://www.findbestopensource.com/tagged/distributed-caching . The caching > system will do LRU and is much more efficient. > > On Wed, Jul 14, 2010 at 12:39 PM, Li Li wrote: > >> I have already store it in lucene index. But it is in disk and When a >> query come, it must seek the disk to get it. I am not familiar with >> lucene cache. I just want to fully use my memory that load 10GB of it >> in memory and a LRU stragety when cache full. To load more into >> memory, I want to compress it "in memory". I don't care much about >> disk space so whether or not it's compressed in lucene . >> >> 2010/7/14 findbestopensource : >> > You have two options >> > 1. Store the compressed text as part of stored field in Solr. >> > 2. Using external caching. >> > http://www.findbestopensource.com/tagged/distributed-caching >> > You could use ehcache / Memcache / Membase. >> > >> > The problem with external caching is you need to synchronize the >> deletions >> > and modification. Fetching the stored field from Solr is also faster. >> > >> > Regards >> > Aditya >> > www.findbestopensource.com >> > >> > >> > On Wed, Jul 14, 2010 at 12:08 PM, Li Li wrote: >> > >> >> I want to cache full text into memory to improve performance. >> >> Full text is only used to highlight in my application(But it's very >> >> time consuming, My avg query time is about 250ms, I guess it will cost >> >> about 50ms if I just get top 10 full text. Things get worse when get >> >> more full text because in disk, it scatters erverywhere for a query.). >> >> My full text per machine is about 200GB. The memory available for >> >> store full text is about 10GB. So I want to compress it in memory. >> >> Suppose compression ratio is 1:5, then I can load 1/4 full text in >> >> memory. I need a Cache component for it. Has anyone faced the problem >> >> before? I need some advice. Is it possbile using external tools such >> >> as MemCached? Thank you. >> >> >> > >> >
Re: Cache full text into memory
I have just provided you two options. Since you already store as part of the index, You could try external caching. Try using ehcache / Membase http://www.findbestopensource.com/tagged/distributed-caching . The caching system will do LRU and is much more efficient. On Wed, Jul 14, 2010 at 12:39 PM, Li Li wrote: > I have already store it in lucene index. But it is in disk and When a > query come, it must seek the disk to get it. I am not familiar with > lucene cache. I just want to fully use my memory that load 10GB of it > in memory and a LRU stragety when cache full. To load more into > memory, I want to compress it "in memory". I don't care much about > disk space so whether or not it's compressed in lucene . > > 2010/7/14 findbestopensource : > > You have two options > > 1. Store the compressed text as part of stored field in Solr. > > 2. Using external caching. > > http://www.findbestopensource.com/tagged/distributed-caching > >You could use ehcache / Memcache / Membase. > > > > The problem with external caching is you need to synchronize the > deletions > > and modification. Fetching the stored field from Solr is also faster. > > > > Regards > > Aditya > > www.findbestopensource.com > > > > > > On Wed, Jul 14, 2010 at 12:08 PM, Li Li wrote: > > > >> I want to cache full text into memory to improve performance. > >> Full text is only used to highlight in my application(But it's very > >> time consuming, My avg query time is about 250ms, I guess it will cost > >> about 50ms if I just get top 10 full text. Things get worse when get > >> more full text because in disk, it scatters erverywhere for a query.). > >> My full text per machine is about 200GB. The memory available for > >> store full text is about 10GB. So I want to compress it in memory. > >> Suppose compression ratio is 1:5, then I can load 1/4 full text in > >> memory. I need a Cache component for it. Has anyone faced the problem > >> before? I need some advice. Is it possbile using external tools such > >> as MemCached? Thank you. > >> > > >
Re: Cache full text into memory
I have already store it in lucene index. But it is in disk and When a query come, it must seek the disk to get it. I am not familiar with lucene cache. I just want to fully use my memory that load 10GB of it in memory and a LRU stragety when cache full. To load more into memory, I want to compress it "in memory". I don't care much about disk space so whether or not it's compressed in lucene . 2010/7/14 findbestopensource : > You have two options > 1. Store the compressed text as part of stored field in Solr. > 2. Using external caching. > http://www.findbestopensource.com/tagged/distributed-caching > You could use ehcache / Memcache / Membase. > > The problem with external caching is you need to synchronize the deletions > and modification. Fetching the stored field from Solr is also faster. > > Regards > Aditya > www.findbestopensource.com > > > On Wed, Jul 14, 2010 at 12:08 PM, Li Li wrote: > >> I want to cache full text into memory to improve performance. >> Full text is only used to highlight in my application(But it's very >> time consuming, My avg query time is about 250ms, I guess it will cost >> about 50ms if I just get top 10 full text. Things get worse when get >> more full text because in disk, it scatters erverywhere for a query.). >> My full text per machine is about 200GB. The memory available for >> store full text is about 10GB. So I want to compress it in memory. >> Suppose compression ratio is 1:5, then I can load 1/4 full text in >> memory. I need a Cache component for it. Has anyone faced the problem >> before? I need some advice. Is it possbile using external tools such >> as MemCached? Thank you. >> >
Re: Cache full text into memory
You have two options 1. Store the compressed text as part of stored field in Solr. 2. Using external caching. http://www.findbestopensource.com/tagged/distributed-caching You could use ehcache / Memcache / Membase. The problem with external caching is you need to synchronize the deletions and modification. Fetching the stored field from Solr is also faster. Regards Aditya www.findbestopensource.com On Wed, Jul 14, 2010 at 12:08 PM, Li Li wrote: > I want to cache full text into memory to improve performance. > Full text is only used to highlight in my application(But it's very > time consuming, My avg query time is about 250ms, I guess it will cost > about 50ms if I just get top 10 full text. Things get worse when get > more full text because in disk, it scatters erverywhere for a query.). > My full text per machine is about 200GB. The memory available for > store full text is about 10GB. So I want to compress it in memory. > Suppose compression ratio is 1:5, then I can load 1/4 full text in > memory. I need a Cache component for it. Has anyone faced the problem > before? I need some advice. Is it possbile using external tools such > as MemCached? Thank you. >
