DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Santanu8939967892
Hi, I have a huge volume of DB records, which is close to 250 millions. I am going to use DIH to index the data into Solr. I need a best architecture to index and query the data in an efficient manner. I am using windows server 2008 with 16 GB RAM, zion processor and Solr 4.4. With Regards, Sa

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Gora Mohanty
On 29 July 2013 17:30, Santanu8939967892 wrote: > Hi, >I have a huge volume of DB records, which is close to 250 millions. > I am going to use DIH to index the data into Solr. > I need a best architecture to index and query the data in an efficient > manner. [...] This is difficult to answer

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Santanu8939967892
Hi Gora, I wanted to highlight one point here. As my content volume is large what should be my index architecture single core, cloud... ? With Regards, Santanu On Mon, Jul 29, 2013 at 5:37 PM, Gora Mohanty wrote: > On 29 July 2013 17:30, Santanu8939967892 wrote: > > Hi, > >I have a h

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Jack Krupansky
data - 250 millions - Need a best architecture Hi, I have a huge volume of DB records, which is close to 250 millions. I am going to use DIH to index the data into Solr. I need a best architecture to index and query the data in an efficient manner. I am using windows server 2008 with 16 GB RAM

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Santanu8939967892
s) will access > data. > > -- Jack Krupansky > > -Original Message- From: Santanu8939967892 > Sent: Monday, July 29, 2013 8:00 AM > To: solr-user@lucene.apache.org > Subject: DIH to index the data - 250 millions - Need a best architecture > > > Hi, > I have a

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Jack Krupansky
29, 2013 9:41 AM To: solr-user@lucene.apache.org Subject: Re: DIH to index the data - 250 millions - Need a best architecture Hi Jack, My sample query will be with a keyword (text) and probably 2 to 3 filters. There is a java interface for display of data, which will consume a class, and the cla

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Shawn Heisey
On 7/29/2013 6:00 AM, Santanu8939967892 wrote: > Hi, >I have a huge volume of DB records, which is close to 250 millions. > I am going to use DIH to index the data into Solr. > I need a best architecture to index and query the data in an efficient > manner. > I am using windows server 2008 with

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Mikhail Khludnev
Mishra, What if you setup DIH with single SQLEntityProcessor without caching, does it works for you? On Mon, Jul 29, 2013 at 4:00 PM, Santanu8939967892 wrote: > Hi, >I have a huge volume of DB records, which is close to 250 millions. > I am going to use DIH to index the data into Solr. > I

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Santanu8939967892
Hi Shawn, Yes, your assumption is correct. The index size is around 250 GB and we index 20/30 meta data and store around 50. We have plan for a Solr cloud architecture having two nodes one Master and other one is replica of the master (replication factor 2) with multiple zookeeper ensembl

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Santanu8939967892
Hi, In addition to my last mail one further query. Can we automate the deployment process for multinode environment (N.. nodes)? With Regards, Santanu On Tue, Jul 30, 2013 at 11:53 AM, Santanu8939967892 < mishra.sant...@gmail.com> wrote: > Hi Shawn, > Yes, your assumption is correct. Th

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-29 Thread Shawn Heisey
On 7/30/2013 12:23 AM, Santanu8939967892 wrote: > Yes, your assumption is correct. The index size is around 250 GB and > we index 20/30 meta data and store around 50. > We have plan for a Solr cloud architecture having two nodes one Master > and other one is replica of the master (replica

Re: DIH to index the data - 250 millions - Need a best architecture

2013-07-30 Thread Santanu8939967892
Hi Shawn, Thanks for your detailed explanation. Will do a POC and finalize the arch. With Regards, Santanu On Tue, Jul 30, 2013 at 12:20 PM, Shawn Heisey wrote: > On 7/30/2013 12:23 AM, Santanu8939967892 wrote: > > Yes, your assumption is correct. The index size is around 250 GB and