Re: Index per user - thousands of indices in one Solr instance
On Jul 23, 2009, at 7:00 AM, Łukasz Osipiuk wrote: See https://issues.apache.org/jira/browse/SOLR-1293 We're planning to put up a patch soon. Perhaps we can collaborate? What are your estimations to have this patches ready. We have quite tight deadlines and cannot afford months of developments. If you are finishing and have some well separated tasks we certainly can help (preferably ones which does not require deep Solr internal understanding). Otherwise we will probably go for a quick hack using lucene directly. I'd strongly recommend you stick with Solr, as you'll be reinventing much more than you might initially imagine by going direct with Lucene. Even without SOLR-1293, Solr (trunk, 1.4 that is) has core administration capability so you can manage cores through an HTTP interface. You can bring new indexes/cores online, and can even with some clever configuration capabilities be able to remotely move/ replicate indexes around if you need to move them to scale separately. You'll be most of the way there already with Solr and what is currently available. Erik
Re: Index per user - thousands of indices in one Solr instance
The patch is up on SOLR-1293. There's a wiki page at http://wiki.apache.org/solr/LotsOfCores with details on the changes and configuration. On Thu, Jul 23, 2009 at 4:51 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > On Thu, Jul 23, 2009 at 4:30 PM, Łukasz Osipiuk wrote: > >> >> > See https://issues.apache.org/jira/browse/SOLR-1293 >> > >> > We're planning to put up a patch soon. Perhaps we can collaborate? >> >> What are your estimations to have this patches ready. We have quite >> tight deadlines >> and cannot afford months of developments. >> If you are finishing and have some well separated tasks we certainly >> can help (preferably ones which does not require deep Solr internal >> understanding). >> Otherwise we will probably go for a quick hack using lucene directly. >> > > It is mostly done with some caveats (some features like alias/unalias are > not supported). We've been doing extensive performance testing with this > patch and we've already seen upto 5x improvement in throughput. > > We'll post the patch by tomorrow so you can take a look and get started. > I'll also start a wiki page and document the various features, configuration > options and performance benchmark results. > > -- > Regards, > Shalin Shekhar Mangar. > -- Regards, Shalin Shekhar Mangar.
Re: Index per user - thousands of indices in one Solr instance
On Thu, Jul 23, 2009 at 4:30 PM, Łukasz Osipiuk wrote: > > > See https://issues.apache.org/jira/browse/SOLR-1293 > > > > We're planning to put up a patch soon. Perhaps we can collaborate? > > What are your estimations to have this patches ready. We have quite > tight deadlines > and cannot afford months of developments. > If you are finishing and have some well separated tasks we certainly > can help (preferably ones which does not require deep Solr internal > understanding). > Otherwise we will probably go for a quick hack using lucene directly. > It is mostly done with some caveats (some features like alias/unalias are not supported). We've been doing extensive performance testing with this patch and we've already seen upto 5x improvement in throughput. We'll post the patch by tomorrow so you can take a look and get started. I'll also start a wiki page and document the various features, configuration options and performance benchmark results. -- Regards, Shalin Shekhar Mangar.
Re: Index per user - thousands of indices in one Solr instance
On Thu, Jul 23, 2009 at 11:44, Shalin Shekhar Mangar wrote: > On Thu, Jul 23, 2009 at 3:06 PM, Łukasz Osipiuk wrote: > >> >> I am new to Solr and I want to get a quick hint if it is suitable for >> what we want to use it for. >> We are building e-mail platform and we want to provide our users with >> full-text search functionality. >> >> We are not willing to use single index file for all users as we want >> to be able to migrate user index from one machine >> to another if need for scaling arises. As we want to have separate >> index file per user, single Solr instance would have to >> handle few thousands (or hundreds of thousands) index files (yet each >> quite small in size). >> We also need to add and remove indices online, as users register >> accounts or are moved to different computer in cluster. >> >> Was Solr designed with such setup in mind? I search the net but did >> not find such usage pattern. >> >> We can directly use Lucene and implement network layer and index >> replication by ourselves but it would be nice to avoid it. >> > > Solr was not designed with such a setup in mind. However, we are working on > a similar use-case and building the additional features Solr would need. Yeah, quite similar :) > See https://issues.apache.org/jira/browse/SOLR-1293 > > We're planning to put up a patch soon. Perhaps we can collaborate? What are your estimations to have this patches ready. We have quite tight deadlines and cannot afford months of developments. If you are finishing and have some well separated tasks we certainly can help (preferably ones which does not require deep Solr internal understanding). Otherwise we will probably go for a quick hack using lucene directly. Regards, Łukasz Osipiuk. -- Łukasz Osipiuk mailto:luk...@osipiuk.net
Re: Index per user - thousands of indices in one Solr instance
On Thu, Jul 23, 2009 at 3:06 PM, Łukasz Osipiuk wrote: > > I am new to Solr and I want to get a quick hint if it is suitable for > what we want to use it for. > We are building e-mail platform and we want to provide our users with > full-text search functionality. > > We are not willing to use single index file for all users as we want > to be able to migrate user index from one machine > to another if need for scaling arises. As we want to have separate > index file per user, single Solr instance would have to > handle few thousands (or hundreds of thousands) index files (yet each > quite small in size). > We also need to add and remove indices online, as users register > accounts or are moved to different computer in cluster. > > Was Solr designed with such setup in mind? I search the net but did > not find such usage pattern. > > We can directly use Lucene and implement network layer and index > replication by ourselves but it would be nice to avoid it. > Solr was not designed with such a setup in mind. However, we are working on a similar use-case and building the additional features Solr would need. See https://issues.apache.org/jira/browse/SOLR-1293 We're planning to put up a patch soon. Perhaps we can collaborate? -- Regards, Shalin Shekhar Mangar.
