Re: RAMDirecotory instead of FSDirectory for SOLR
Hi Jeryl, Three weeks later - any luck with Solr + Terracotta? Thanks, Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Jeryl Cook <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Friday, June 1, 2007 3:59:21 AM Subject: RE: RAMDirecotory instead of FSDirectory for SOLR i have Terracotta to work with Lucene , and it works find with the RAMDirectory...i am trying to get it to work with SOLR(Hook the RAMDirectory..)..., when i do, ill post the findings,problems,etc..Thanks for feedback from everyone.Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)> Date: Thu, 31 May 2007 18:24:26 -0700> From: [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE: RAMDirecotory instead of FSDirectory for SOLR> > > Jeryl,> > If you need any help getting Terracotta to work under Lucene or if you have> any questions about performance tuning and/or load testing, you can also use> the Terracotta community resources (mailing lists, forums, IRC, whatnot):> <http://www.terracotta.org/confluence/display/orgsite/Community>. We'd be> more than happy to help you get this stuff working.> > Cheers,> Orion> > > Jeryl Cook wrote:> > > > Thats the thing,Terracotta persists everything it has in memory to the> > disk when it overflows(u can set how much u want to use in memory), or> > when the server goes offline. When the server comes back the master> > terracotta simply loads it back into the memory of the once offline> > worker..identical to the approach SOLR already does to handle> > scalability, this allows unlimited storage of the items in memory, ...> > you just need to cluster the "RAMDirectory" according to the sample giving> > by TerracottaHowever i read some of the post here...I read some say: "> > i wonder how performance will be".,etci was trying to get it> > working..andload test the hell out it, and see how it acts with large> > amounts of data, and how it ompares with SOLR using typical FSDirectory> > approach.i plan to post findings..Jeryl Cook > > > > > > > > /^\ Pharaoh /^\ > > > > http://pharaohofkush.blogspot.com/ > > > > > > > > "..Act your age, and not your shoe size.."> > > > -Prince(1986)> Date: Thu, 31 May 2007 13:51:53 -0700> From:> > [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE:> > RAMDirecotory instead of FSDirectory for SOLR> > > : board, looks like i> > can achieve this with the "embedded" version of SOLR> : uses the lucene> > RAMDirectory to store the index..Jeryl Cook> > yeah ... adding> > asolrconfig.xml option for using a RAMDirectory would be> possible ... but> > almost meaningless for most people (the directory would> go away when the> > server shuts down) ... even for use cases like what you> describe (hooking> > in terrecota) it wouldn't be enough in itself, because> there would be no> > hook to give terracota access to it.> > > -Hoss> > > > > -- > View this message in context: http://www.nabble.com/RAMDirecotory-instead-of-FSDirectory-for-SOLR-tf3843377.html#a10905062> Sent from the Solr - User mailing list archive at Nabble.com.>
RE: RAMDirecotory instead of FSDirectory for SOLR
i have Terracotta to work with Lucene , and it works find with the RAMDirectory...i am trying to get it to work with SOLR(Hook the RAMDirectory..)..., when i do, ill post the findings,problems,etc..Thanks for feedback from everyone.Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)> Date: Thu, 31 May 2007 18:24:26 -0700> From: [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE: RAMDirecotory instead of FSDirectory for SOLR> > > Jeryl,> > If you need any help getting Terracotta to work under Lucene or if you have> any questions about performance tuning and/or load testing, you can also use> the Terracotta community resources (mailing lists, forums, IRC, whatnot):> <http://www.terracotta.org/confluence/display/orgsite/Community>. We'd be> more than happy to help you get this stuff working.> > Cheers,> Orion> > > Jeryl Cook wrote:> > > > Thats the thing,Terracotta persists everything it has in memory to the> > disk when it overflows(u can set how much u want to use in memory), or> > when the server goes offline. When the server comes back the master> > terracotta simply loads it back into the memory of the once offline> > worker..identical to the approach SOLR already does to handle> > scalability, this allows unlimited storage of the items in memory, ...> > you just need to cluster the "RAMDirectory" according to the sample giving> > by TerracottaHowever i read some of the post here...I read some say: "> > i wonder how performance will be".,etci was trying to get it> > working..andload test the hell out it, and see how it acts with large> > amounts of data, and how it ompares with SOLR using typical FSDirectory> > approach.i plan to post findings..Jeryl Cook > > > > > > > > /^\ Pharaoh /^\ > > > > http://pharaohofkush.blogspot.com/ > > > > > > > > "..Act your age, and not your shoe size.."> > > > -Prince(1986)> Date: Thu, 31 May 2007 13:51:53 -0700> From:> > [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE:> > RAMDirecotory instead of FSDirectory for SOLR> > > : board, looks like i> > can achieve this with the "embedded" version of SOLR> : uses the lucene> > RAMDirectory to store the index..Jeryl Cook> > yeah ... adding> > asolrconfig.xml option for using a RAMDirectory would be> possible ... but> > almost meaningless for most people (the directory would> go away when the> > server shuts down) ... even for use cases like what you> describe (hooking> > in terrecota) it wouldn't be enough in itself, because> there would be no> > hook to give terracota access to it.> > > -Hoss> > > > > -- > View this message in context: http://www.nabble.com/RAMDirecotory-instead-of-FSDirectory-for-SOLR-tf3843377.html#a10905062> Sent from the Solr - User mailing list archive at Nabble.com.>
RE: RAMDirecotory instead of FSDirectory for SOLR
Jeryl, If you need any help getting Terracotta to work under Lucene or if you have any questions about performance tuning and/or load testing, you can also use the Terracotta community resources (mailing lists, forums, IRC, whatnot): <http://www.terracotta.org/confluence/display/orgsite/Community>. We'd be more than happy to help you get this stuff working. Cheers, Orion Jeryl Cook wrote: > > Thats the thing,Terracotta persists everything it has in memory to the > disk when it overflows(u can set how much u want to use in memory), or > when the server goes offline. When the server comes back the master > terracotta simply loads it back into the memory of the once offline > worker..identical to the approach SOLR already does to handle > scalability, this allows unlimited storage of the items in memory, ... > you just need to cluster the "RAMDirectory" according to the sample giving > by TerracottaHowever i read some of the post here...I read some say: " > i wonder how performance will be".,etci was trying to get it > working..andload test the hell out it, and see how it acts with large > amounts of data, and how it ompares with SOLR using typical FSDirectory > approach.i plan to post findings..Jeryl Cook > > > > /^\ Pharaoh /^\ > > http://pharaohofkush.blogspot.com/ > > > > "..Act your age, and not your shoe size.." > > -Prince(1986)> Date: Thu, 31 May 2007 13:51:53 -0700> From: > [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE: > RAMDirecotory instead of FSDirectory for SOLR> > > : board, looks like i > can achieve this with the "embedded" version of SOLR> : uses the lucene > RAMDirectory to store the index..Jeryl Cook> > yeah ... adding > asolrconfig.xml option for using a RAMDirectory would be> possible ... but > almost meaningless for most people (the directory would> go away when the > server shuts down) ... even for use cases like what you> describe (hooking > in terrecota) it wouldn't be enough in itself, because> there would be no > hook to give terracota access to it.> > > -Hoss> > -- View this message in context: http://www.nabble.com/RAMDirecotory-instead-of-FSDirectory-for-SOLR-tf3843377.html#a10905062 Sent from the Solr - User mailing list archive at Nabble.com.
RE: RAMDirecotory instead of FSDirectory for SOLR
: Thats the thing,Terracotta persists everything it has in memory to the : disk when it overflows(u can set how much u want to use in memory), or : when the server goes offline. When the server comes back the master : terracotta simply loads it back into the memory of the once offline Sure .. but Terracotta needs some control over the RAMDirectory to do all this ... i'm also assuming that for the "loading back into memory" part to work when a server starts up, terracotta actually needs to *own* the RAMDirectory, so just making solr create a RAMDirectory on startup based on a config option wouldn't be enough ... there would need to be some (probably complex) hooks to determine how that RAMDirectory gets created/managed to make what you are describing work. -Hoss
RE: RAMDirecotory instead of FSDirectory for SOLR
Thats the thing,Terracotta persists everything it has in memory to the disk when it overflows(u can set how much u want to use in memory), or when the server goes offline. When the server comes back the master terracotta simply loads it back into the memory of the once offline worker..identical to the approach SOLR already does to handle scalability, this allows unlimited storage of the items in memory, ... you just need to cluster the "RAMDirectory" according to the sample giving by TerracottaHowever i read some of the post here...I read some say: " i wonder how performance will be".,etci was trying to get it working..andload test the hell out it, and see how it acts with large amounts of data, and how it ompares with SOLR using typical FSDirectory approach.i plan to post findings..Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)> Date: Thu, 31 May 2007 13:51:53 -0700> From: [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE: RAMDirecotory instead of FSDirectory for SOLR> > > : board, looks like i can achieve this with the "embedded" version of SOLR> : uses the lucene RAMDirectory to store the index..Jeryl Cook> > yeah ... adding asolrconfig.xml option for using a RAMDirectory would be> possible ... but almost meaningless for most people (the directory would> go away when the server shuts down) ... even for use cases like what you> describe (hooking in terrecota) it wouldn't be enough in itself, because> there would be no hook to give terracota access to it.> > > -Hoss>
RE: RAMDirecotory instead of FSDirectory for SOLR
: board, looks like i can achieve this with the "embedded" version of SOLR : uses the lucene RAMDirectory to store the index..Jeryl Cook yeah ... adding asolrconfig.xml option for using a RAMDirectory would be possible ... but almost meaningless for most people (the directory would go away when the server shuts down) ... even for use cases like what you describe (hooking in terrecota) it wouldn't be enough in itself, because there would be no hook to give terracota access to it. -Hoss
RE: RAMDirecotory instead of FSDirectory for SOLR
Thanks for the responses (smile..)...actually looking over the message board, looks like i can achieve this with the "embedded" version of SOLR uses the lucene RAMDirectory to store the index..Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)From: [EMAIL PROTECTED]: [EMAIL PROTECTED]: RAMDirecotory instead of FSDirectory for SOLRDate: Wed, 30 May 2007 18:31:04 -0400 Is it possible to simply change configuration to use RAMDirectory , instead of the FSDirectory..if not it would be great to have this as possible option int he configuration fileThe Master/Worker pattern used for handling scalability works(outlined in SOLR manual/wiki).its a proven pattern..however Terracotta , http://terracottatech.com/ is able to cluster the RAMDirectory(items that cannot fit in memory are managed written to disk.) ..i would love to take advantageof this approach...can you tell me if it is possible to switch out??Thanks.Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)
RAMDirecotory instead of FSDirectory for SOLR
Is it possible to simply change configuration to use RAMDirectory , instead of the FSDirectory..if not it would be great to have this as possible option int he configuration fileThe Master/Worker pattern used for handling scalability works(outlined in SOLR manual/wiki).its a proven pattern..however Terracotta , http://terracottatech.com/ is able to cluster the RAMDirectory(items that cannot fit in memory are managed written to disk.) ..i would love to take advantageof this approach...can you tell me if it is possible to switch out??Thanks.Jeryl Cook /^\ Pharaoh /^\ http://pharaohofkush.blogspot.com/ "..Act your age, and not your shoe size.." -Prince(1986)
