Re: Storing different entities in Solr
Thanks for the replies guys. I am not at work so I don't have the exact schema but here's what it roughly looks like: Request: == id client_id pm_id pm2_id title Advisor: == id person_id address_id bio sector (IT, doctor, etc) There's another table. RequestAdvisor: = id advisor_id request_id The idea of adding a prefix to primary keys does sound good. I can just do advisor_123 and request_12345. On Sun, May 30, 2010 at 9:22 PM, Bill Au wrote: > There is only one primary key in a single index. If the id of your > different document types do collide, you can simply add a prefix or suffix > to make them unique. > > Bill > > On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan wrote: > >> Thanks for all your answers guys. Requests and consultants have a many >> to many relationship so I can't store request info in a document with >> advisorID as the primary key. >> >> Bill's solution and multicore solutions might be what I am looking >> for. Bill, will I be able to have 2 primary keys (so I can update and >> delete documents)? If yes, can you please give me a link or someting >> where I can get more info on this? >> >> Thanks, >> Moazzam >> >> >> >> On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: >> > You can keep different type of documents in the same index. If each >> > document has a type field. You can restrict your searches to specific >> > type(s) of document by using a filter query, which is very fast and >> > efficient. >> > >> > Bill >> > >> > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < >> > knagelb...@globeandmail.com> wrote: >> > >> >> Multi-core is an option, but keep in mind if you go that route you will >> >> need to do two searches to correlate data between the two. >> >> >> >> -Kallin Nagelberg >> >> >> >> -Original Message- >> >> From: Robert Zotter [mailto:robertzot...@gmail.com] >> >> Sent: Friday, May 28, 2010 12:26 PM >> >> To: solr-user@lucene.apache.org >> >> Subject: Re: Storing different entities in Solr >> >> >> >> >> >> Sounds like you'll want to use a multiple core setup. One core fore each >> >> type >> >> of "document" >> >> >> >> http://wiki.apache.org/solr/CoreAdmin >> >> -- >> >> View this message in context: >> >> >> http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html >> >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> >> > >> >
Re: Storing different entities in Solr
There is only one primary key in a single index. If the id of your different document types do collide, you can simply add a prefix or suffix to make them unique. Bill On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan wrote: > Thanks for all your answers guys. Requests and consultants have a many > to many relationship so I can't store request info in a document with > advisorID as the primary key. > > Bill's solution and multicore solutions might be what I am looking > for. Bill, will I be able to have 2 primary keys (so I can update and > delete documents)? If yes, can you please give me a link or someting > where I can get more info on this? > > Thanks, > Moazzam > > > > On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: > > You can keep different type of documents in the same index. If each > > document has a type field. You can restrict your searches to specific > > type(s) of document by using a filter query, which is very fast and > > efficient. > > > > Bill > > > > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < > > knagelb...@globeandmail.com> wrote: > > > >> Multi-core is an option, but keep in mind if you go that route you will > >> need to do two searches to correlate data between the two. > >> > >> -Kallin Nagelberg > >> > >> -----Original Message----- > >> From: Robert Zotter [mailto:robertzot...@gmail.com] > >> Sent: Friday, May 28, 2010 12:26 PM > >> To: solr-user@lucene.apache.org > >> Subject: Re: Storing different entities in Solr > >> > >> > >> Sounds like you'll want to use a multiple core setup. One core fore each > >> type > >> of "document" > >> > >> http://wiki.apache.org/solr/CoreAdmin > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > > >
Re: Storing different entities in Solr
The size of the join table is the number of documents, if you denormalize the two tables. On Fri, May 28, 2010 at 11:38 AM, Nagelberg, Kallin wrote: > I agree with Erick, > > Could you show us what these two entities look like, and the total count of > each? That might shed some light on the appropriate approach. > > -Kallin Nagelberg > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, May 28, 2010 2:36 PM > To: solr-user@lucene.apache.org > Subject: Re: Storing different entities in Solr > > You most certainly *can* store the many<->many relationship, you > are just denormalizing your data. I know it goes against the grain > of any good database admin, but it's very often a good solution > for a search application. > > You've gotta forget almost everything you learned about how data > *should* be stored in databases when working with a search app. > Well, perhaps I'm overstating a bit, but you get the idea > > When I see messages about primary keys and foreign keys etc, I > break out in hives. It's almost always a mistake to try to force > lucene/solr to behave like a database. Whenever you find yourself > trying, stop, take a deep breath, and think about searching ... > > A lot depends on how much data we're talking about here. If > fully denormalizing things would cost you 10M, who cares? If it > would cost you 100G, it's a different story > > Best > Erick > > > On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan wrote: > >> Thanks for all your answers guys. Requests and consultants have a many >> to many relationship so I can't store request info in a document with >> advisorID as the primary key. >> >> Bill's solution and multicore solutions might be what I am looking >> for. Bill, will I be able to have 2 primary keys (so I can update and >> delete documents)? If yes, can you please give me a link or someting >> where I can get more info on this? >> >> Thanks, >> Moazzam >> >> >> >> On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: >> > You can keep different type of documents in the same index. If each >> > document has a type field. You can restrict your searches to specific >> > type(s) of document by using a filter query, which is very fast and >> > efficient. >> > >> > Bill >> > >> > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < >> > knagelb...@globeandmail.com> wrote: >> > >> >> Multi-core is an option, but keep in mind if you go that route you will >> >> need to do two searches to correlate data between the two. >> >> >> >> -Kallin Nagelberg >> >> >> >> -Original Message- >> >> From: Robert Zotter [mailto:robertzot...@gmail.com] >> >> Sent: Friday, May 28, 2010 12:26 PM >> >> To: solr-user@lucene.apache.org >> >> Subject: Re: Storing different entities in Solr >> >> >> >> >> >> Sounds like you'll want to use a multiple core setup. One core fore each >> >> type >> >> of "document" >> >> >> >> http://wiki.apache.org/solr/CoreAdmin >> >> -- >> >> View this message in context: >> >> >> http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html >> >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> >> > >> > -- Lance Norskog goks...@gmail.com
RE: Storing different entities in Solr
I agree with Erick, Could you show us what these two entities look like, and the total count of each? That might shed some light on the appropriate approach. -Kallin Nagelberg -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, May 28, 2010 2:36 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr You most certainly *can* store the many<->many relationship, you are just denormalizing your data. I know it goes against the grain of any good database admin, but it's very often a good solution for a search application. You've gotta forget almost everything you learned about how data *should* be stored in databases when working with a search app. Well, perhaps I'm overstating a bit, but you get the idea When I see messages about primary keys and foreign keys etc, I break out in hives. It's almost always a mistake to try to force lucene/solr to behave like a database. Whenever you find yourself trying, stop, take a deep breath, and think about searching ... A lot depends on how much data we're talking about here. If fully denormalizing things would cost you 10M, who cares? If it would cost you 100G, it's a different story Best Erick On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan wrote: > Thanks for all your answers guys. Requests and consultants have a many > to many relationship so I can't store request info in a document with > advisorID as the primary key. > > Bill's solution and multicore solutions might be what I am looking > for. Bill, will I be able to have 2 primary keys (so I can update and > delete documents)? If yes, can you please give me a link or someting > where I can get more info on this? > > Thanks, > Moazzam > > > > On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: > > You can keep different type of documents in the same index. If each > > document has a type field. You can restrict your searches to specific > > type(s) of document by using a filter query, which is very fast and > > efficient. > > > > Bill > > > > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < > > knagelb...@globeandmail.com> wrote: > > > >> Multi-core is an option, but keep in mind if you go that route you will > >> need to do two searches to correlate data between the two. > >> > >> -Kallin Nagelberg > >> > >> -----Original Message- > >> From: Robert Zotter [mailto:robertzot...@gmail.com] > >> Sent: Friday, May 28, 2010 12:26 PM > >> To: solr-user@lucene.apache.org > >> Subject: Re: Storing different entities in Solr > >> > >> > >> Sounds like you'll want to use a multiple core setup. One core fore each > >> type > >> of "document" > >> > >> http://wiki.apache.org/solr/CoreAdmin > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > > >
Re: Storing different entities in Solr
You most certainly *can* store the many<->many relationship, you are just denormalizing your data. I know it goes against the grain of any good database admin, but it's very often a good solution for a search application. You've gotta forget almost everything you learned about how data *should* be stored in databases when working with a search app. Well, perhaps I'm overstating a bit, but you get the idea When I see messages about primary keys and foreign keys etc, I break out in hives. It's almost always a mistake to try to force lucene/solr to behave like a database. Whenever you find yourself trying, stop, take a deep breath, and think about searching ... A lot depends on how much data we're talking about here. If fully denormalizing things would cost you 10M, who cares? If it would cost you 100G, it's a different story Best Erick On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan wrote: > Thanks for all your answers guys. Requests and consultants have a many > to many relationship so I can't store request info in a document with > advisorID as the primary key. > > Bill's solution and multicore solutions might be what I am looking > for. Bill, will I be able to have 2 primary keys (so I can update and > delete documents)? If yes, can you please give me a link or someting > where I can get more info on this? > > Thanks, > Moazzam > > > > On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: > > You can keep different type of documents in the same index. If each > > document has a type field. You can restrict your searches to specific > > type(s) of document by using a filter query, which is very fast and > > efficient. > > > > Bill > > > > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < > > knagelb...@globeandmail.com> wrote: > > > >> Multi-core is an option, but keep in mind if you go that route you will > >> need to do two searches to correlate data between the two. > >> > >> -Kallin Nagelberg > >> > >> -----Original Message- > >> From: Robert Zotter [mailto:robertzot...@gmail.com] > >> Sent: Friday, May 28, 2010 12:26 PM > >> To: solr-user@lucene.apache.org > >> Subject: Re: Storing different entities in Solr > >> > >> > >> Sounds like you'll want to use a multiple core setup. One core fore each > >> type > >> of "document" > >> > >> http://wiki.apache.org/solr/CoreAdmin > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > > >
Re: Storing different entities in Solr
Hi, So for your use case are you wanting to search for a consultant then look at all of his or her request or pull both at the same time? In both cases one index should suffice. In you define a primary key field and use it for both doc types it shouldn't be an issue. Unless your dataset in very large it would reduce the overhead of running a multicore solution especially in indexing etc David Stuart On 28 May 2010, at 18:12, Moazzam Khan wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < knagelb...@globeandmail.com> wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of "document" http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au wrote: > You can keep different type of documents in the same index. If each > document has a type field. You can restrict your searches to specific > type(s) of document by using a filter query, which is very fast and > efficient. > > Bill > > On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < > knagelb...@globeandmail.com> wrote: > >> Multi-core is an option, but keep in mind if you go that route you will >> need to do two searches to correlate data between the two. >> >> -Kallin Nagelberg >> >> -Original Message- >> From: Robert Zotter [mailto:robertzot...@gmail.com] >> Sent: Friday, May 28, 2010 12:26 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Storing different entities in Solr >> >> >> Sounds like you'll want to use a multiple core setup. One core fore each >> type >> of "document" >> >> http://wiki.apache.org/solr/CoreAdmin >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >
Re: Storing different entities in Solr
You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin < knagelb...@globeandmail.com> wrote: > Multi-core is an option, but keep in mind if you go that route you will > need to do two searches to correlate data between the two. > > -Kallin Nagelberg > > -Original Message- > From: Robert Zotter [mailto:robertzot...@gmail.com] > Sent: Friday, May 28, 2010 12:26 PM > To: solr-user@lucene.apache.org > Subject: Re: Storing different entities in Solr > > > Sounds like you'll want to use a multiple core setup. One core fore each > type > of "document" > > http://wiki.apache.org/solr/CoreAdmin > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html > Sent from the Solr - User mailing list archive at Nabble.com. >
RE: Storing different entities in Solr
Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of "document" http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
Sounds like you'll want to use a multiple core setup. One core fore each type of "document" http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Storing different entities in Solr
Good read here: http://mysolr.com/tips/denormalized-data-structure/ . Are consultation requests unique to each consultant? In that case you could represent the request as a Json String and store it as a multi-valued string field for each consultant, though that makes querying against requests trickier. If you need to search against specific fields in the consultant requests than you could try a schema where the consultant is your primary entity and have fields like consultantrequests-field1, consultantrequests-field2, consultantrequests-field3 and then one consultantrequests-fulljson all multi-valued. You could query against the specific fields, then associate to the whole request by searching the json object. It's an approach I've used with success. -Kallin Nagelberg -Original Message- From: Moazzam Khan [mailto:moazz...@gmail.com] Sent: Friday, May 28, 2010 12:17 PM To: solr-user@lucene.apache.org Subject: Storing different entities in Solr Hi Guys, Is there a way to store 2 types of things in Solr. We have a list of consultants and a list of consultation requests. and I want to store them as separate documents. Can I do this with one instance of Solr or do I have to have two instances? Thanks, MOazzam
