Re: Multiple Word Facets
On Oct 27, 2010, at 6:29am, Adam Estrada wrote: Ahhh...I see! I am doing my testing crawling a couple websites using Nutch and in doing so I am assigning my facets to the title field which is type=text. Are you saying that I will need to manually generate the content for my facet field? I can see the reason and need for doing it that way but I really need for my faceting to happen dynamically based on the content in the field which in this case is the title of a URL. You would use copyfield to copy the contents of the title into a new field that uses the string type, and is the one you use for faceting. -- Ken On Wed, Oct 27, 2010 at 9:19 AM, Jayendra Patil wrote: The Shingle Filter Breaks the words in a sentence into a combination of 2/3 words. For faceting field you should use :- stored="true" multiValued="true"/> The type of the field should be *string *so that it is not tokenised at all. On Wed, Oct 27, 2010 at 9:12 AM, Adam Estrada wrote: Thanks guys, the solr.ShingleFilterFactory did work to get me multiple terms per facet but now I am seeing some redundancy in the facets numbers. See below... Highway (62) Highway System (59) National (59) National Highway (59) National Highway System (59) System (59) See what's going on here? How can I make my multi token facets smarter so that the tokens aren't duplicated? Thanks in advance, Adam On Tue, Oct 26, 2010 at 10:32 PM, Ahmet Arslan wrote: Facets are generated from indexed terms. Depending on your need/use-case: You can use a additional separate String field (which is not tokenized) for facets, populate it via copyField. Search on tokenized field facet on non-tokenized field. Or You can add solr.ShingleFilterFactory to your index analyzer to form multiple word terms. --- On Wed, 10/27/10, Adam Estrada wrote: From: Adam Estrada Subject: Multiple Word Facets To: solr-user@lucene.apache.org Date: Wednesday, October 27, 2010, 4:43 AM All, I am a new to Solr faceting and stuck on how to get multiple-word facets returned from a standard Solr query. See below for what is currently being returned. 89 87 87 87 84 60 32 22 19 15 15 14 12 11 10 9 7 7 7 6 6 6 6 ...etc... There are many terms in there that are 2 or 3 word phrases. For example, Eastern Federal Lands Highway Division all gets broken down in to the individual words that make up the total group of words. I've seen quite a few websites that do what it is I am trying to do here so any suggestions at this point would be great. See my schema below (copied from the example schema). Similar for type="query". Please advise on how to group or cluster document terms so that they can be used as facets. Many thanks in advance, Adam Estrada -- Ken Krugler +1 530-210-6378 http://bixolabs.com e l a s t i c w e b m i n i n g
Re: Multiple Word Facets
Ahhh...I see! I am doing my testing crawling a couple websites using Nutch and in doing so I am assigning my facets to the title field which is type=text. Are you saying that I will need to manually generate the content for my facet field? I can see the reason and need for doing it that way but I really need for my faceting to happen dynamically based on the content in the field which in this case is the title of a URL. Thanks again for all the tips on getting this working for me. Adam On Wed, Oct 27, 2010 at 9:19 AM, Jayendra Patil wrote: > The Shingle Filter Breaks the words in a sentence into a combination of 2/3 > words. > > For faceting field you should use :- > multiValued="true"/> > > The type of the field should be *string *so that it is not tokenised at all. > > On Wed, Oct 27, 2010 at 9:12 AM, Adam Estrada wrote: > >> Thanks guys, the solr.ShingleFilterFactory did work to get me multiple >> terms per facet but now I am seeing some redundancy in the facets >> numbers. See below... >> >> Highway (62) >> Highway System (59) >> National (59) >> National Highway (59) >> National Highway System (59) >> System (59) >> >> See what's going on here? How can I make my multi token facets smarter >> so that the tokens aren't duplicated? >> >> Thanks in advance, >> Adam >> >> On Tue, Oct 26, 2010 at 10:32 PM, Ahmet Arslan wrote: >> > Facets are generated from indexed terms. >> > >> > Depending on your need/use-case: >> > >> > You can use a additional separate String field (which is not tokenized) >> for facets, populate it via copyField. Search on tokenized field facet on >> non-tokenized field. >> > >> > Or >> > >> > You can add solr.ShingleFilterFactory to your index analyzer to form >> multiple word terms. >> > >> > --- On Wed, 10/27/10, Adam Estrada wrote: >> > >> >> From: Adam Estrada >> >> Subject: Multiple Word Facets >> >> To: solr-user@lucene.apache.org >> >> Date: Wednesday, October 27, 2010, 4:43 AM >> >> All, >> >> I am a new to Solr faceting and stuck on how to get >> >> multiple-word >> >> facets returned from a standard Solr query. See below for >> >> what is >> >> currently being returned. >> >> >> >> >> >> >> >> >> >> >> >> 89 >> >> 87 >> >> 87 >> >> 87 >> >> 84 >> >> 60 >> >> 32 >> >> 22 >> >> 19 >> >> 15 >> >> 15 >> >> 14 >> >> 12 >> >> 11 >> >> 10 >> >> 9 >> >> 7 >> >> 7 >> >> 7 >> >> 6 >> >> 6 >> >> 6 >> >> 6 >> >> ...etc... >> >> >> >> There are many terms in there that are 2 or 3 word phrases. >> >> For >> >> example, Eastern Federal Lands Highway Division all gets >> >> broken down >> >> in to the individual words that make up the total group of >> >> words. I've >> >> seen quite a few websites that do what it is I am trying to >> >> do here so >> >> any suggestions at this point would be great. See my schema >> >> below >> >> (copied from the example schema). >> >> >> >> > >> class="solr.TextField" positionIncrementGap="100"> >> >> >> >> > >> class="solr.WhitespaceTokenizerFactory"/> >> >> > >> class="solr.SynonymFilterFactory" synonyms="synonyms.txt" >> >> ignoreCase="true" expand="false"/> >> >> > >> class="solr.StopFilterFactory" >> >> >> >> ignoreCase="true" >> >> >> >> words="stopwords.txt" >> >> >> >> enablePositionIncrements="true" >> >> >> >> /> >> >> > >> class="solr.WordDelimiterFilterFactory" >> >> generateWordParts="1" >> >> generateNumberParts="1" catenateWords="0" >> >> catenateNumbers="0" >> >> catenateAll="0" splitOnCaseChange="1"/> >> >> > >> class="solr.RemoveDuplicatesTokenFilterFactory"/> >> >> >> >> >> >> Similar for type="query". Please advise on how to group or >> >> cluster >> >> document terms so that they can be used as facets. >> >> >> >> Many thanks in advance, >> >> Adam Estrada >> >> >> > >> > >> > >> > >> >
Re: Multiple Word Facets
The Shingle Filter Breaks the words in a sentence into a combination of 2/3 words. For faceting field you should use :- The type of the field should be *string *so that it is not tokenised at all. On Wed, Oct 27, 2010 at 9:12 AM, Adam Estrada wrote: > Thanks guys, the solr.ShingleFilterFactory did work to get me multiple > terms per facet but now I am seeing some redundancy in the facets > numbers. See below... > > Highway (62) > Highway System (59) > National (59) > National Highway (59) > National Highway System (59) > System (59) > > See what's going on here? How can I make my multi token facets smarter > so that the tokens aren't duplicated? > > Thanks in advance, > Adam > > On Tue, Oct 26, 2010 at 10:32 PM, Ahmet Arslan wrote: > > Facets are generated from indexed terms. > > > > Depending on your need/use-case: > > > > You can use a additional separate String field (which is not tokenized) > for facets, populate it via copyField. Search on tokenized field facet on > non-tokenized field. > > > > Or > > > > You can add solr.ShingleFilterFactory to your index analyzer to form > multiple word terms. > > > > --- On Wed, 10/27/10, Adam Estrada wrote: > > > >> From: Adam Estrada > >> Subject: Multiple Word Facets > >> To: solr-user@lucene.apache.org > >> Date: Wednesday, October 27, 2010, 4:43 AM > >> All, > >> I am a new to Solr faceting and stuck on how to get > >> multiple-word > >> facets returned from a standard Solr query. See below for > >> what is > >> currently being returned. > >> > >> > >> > >> > >> > >> 89 > >> 87 > >> 87 > >> 87 > >> 84 > >> 60 > >> 32 > >> 22 > >> 19 > >> 15 > >> 15 > >> 14 > >> 12 > >> 11 > >> 10 > >> 9 > >> 7 > >> 7 > >> 7 > >> 6 > >> 6 > >> 6 > >> 6 > >> ...etc... > >> > >> There are many terms in there that are 2 or 3 word phrases. > >> For > >> example, Eastern Federal Lands Highway Division all gets > >> broken down > >> in to the individual words that make up the total group of > >> words. I've > >> seen quite a few websites that do what it is I am trying to > >> do here so > >> any suggestions at this point would be great. See my schema > >> below > >> (copied from the example schema). > >> > >> >> class="solr.TextField" positionIncrementGap="100"> > >> > >> >> class="solr.WhitespaceTokenizerFactory"/> > >> >> class="solr.SynonymFilterFactory" synonyms="synonyms.txt" > >> ignoreCase="true" expand="false"/> > >> >> class="solr.StopFilterFactory" > >> > >> ignoreCase="true" > >> > >> words="stopwords.txt" > >> > >> enablePositionIncrements="true" > >> > >> /> > >> >> class="solr.WordDelimiterFilterFactory" > >> generateWordParts="1" > >> generateNumberParts="1" catenateWords="0" > >> catenateNumbers="0" > >> catenateAll="0" splitOnCaseChange="1"/> > >> >> class="solr.RemoveDuplicatesTokenFilterFactory"/> > >> > >> > >> Similar for type="query". Please advise on how to group or > >> cluster > >> document terms so that they can be used as facets. > >> > >> Many thanks in advance, > >> Adam Estrada > >> > > > > > > > > >
Re: Multiple Word Facets
Thanks guys, the solr.ShingleFilterFactory did work to get me multiple terms per facet but now I am seeing some redundancy in the facets numbers. See below... Highway (62) Highway System (59) National (59) National Highway (59) National Highway System (59) System (59) See what's going on here? How can I make my multi token facets smarter so that the tokens aren't duplicated? Thanks in advance, Adam On Tue, Oct 26, 2010 at 10:32 PM, Ahmet Arslan wrote: > Facets are generated from indexed terms. > > Depending on your need/use-case: > > You can use a additional separate String field (which is not tokenized) for > facets, populate it via copyField. Search on tokenized field facet on > non-tokenized field. > > Or > > You can add solr.ShingleFilterFactory to your index analyzer to form multiple > word terms. > > --- On Wed, 10/27/10, Adam Estrada wrote: > >> From: Adam Estrada >> Subject: Multiple Word Facets >> To: solr-user@lucene.apache.org >> Date: Wednesday, October 27, 2010, 4:43 AM >> All, >> I am a new to Solr faceting and stuck on how to get >> multiple-word >> facets returned from a standard Solr query. See below for >> what is >> currently being returned. >> >> >> >> >> >> 89 >> 87 >> 87 >> 87 >> 84 >> 60 >> 32 >> 22 >> 19 >> 15 >> 15 >> 14 >> 12 >> 11 >> 10 >> 9 >> 7 >> 7 >> 7 >> 6 >> 6 >> 6 >> 6 >> ...etc... >> >> There are many terms in there that are 2 or 3 word phrases. >> For >> example, Eastern Federal Lands Highway Division all gets >> broken down >> in to the individual words that make up the total group of >> words. I've >> seen quite a few websites that do what it is I am trying to >> do here so >> any suggestions at this point would be great. See my schema >> below >> (copied from the example schema). >> >> > class="solr.TextField" positionIncrementGap="100"> >> >> > class="solr.WhitespaceTokenizerFactory"/> >> > class="solr.SynonymFilterFactory" synonyms="synonyms.txt" >> ignoreCase="true" expand="false"/> >> > class="solr.StopFilterFactory" >> >> ignoreCase="true" >> >> words="stopwords.txt" >> >> enablePositionIncrements="true" >> >> /> >> > class="solr.WordDelimiterFilterFactory" >> generateWordParts="1" >> generateNumberParts="1" catenateWords="0" >> catenateNumbers="0" >> catenateAll="0" splitOnCaseChange="1"/> >> > class="solr.RemoveDuplicatesTokenFilterFactory"/> >> >> >> Similar for type="query". Please advise on how to group or >> cluster >> document terms so that they can be used as facets. >> >> Many thanks in advance, >> Adam Estrada >> > > > >
Re: Multiple Word Facets
Facets are generated from indexed terms. Depending on your need/use-case: You can use a additional separate String field (which is not tokenized) for facets, populate it via copyField. Search on tokenized field facet on non-tokenized field. Or You can add solr.ShingleFilterFactory to your index analyzer to form multiple word terms. --- On Wed, 10/27/10, Adam Estrada wrote: > From: Adam Estrada > Subject: Multiple Word Facets > To: solr-user@lucene.apache.org > Date: Wednesday, October 27, 2010, 4:43 AM > All, > I am a new to Solr faceting and stuck on how to get > multiple-word > facets returned from a standard Solr query. See below for > what is > currently being returned. > > > > > > 89 > 87 > 87 > 87 > 84 > 60 > 32 > 22 > 19 > 15 > 15 > 14 > 12 > 11 > 10 > 9 > 7 > 7 > 7 > 6 > 6 > 6 > 6 > ...etc... > > There are many terms in there that are 2 or 3 word phrases. > For > example, Eastern Federal Lands Highway Division all gets > broken down > in to the individual words that make up the total group of > words. I've > seen quite a few websites that do what it is I am trying to > do here so > any suggestions at this point would be great. See my schema > below > (copied from the example schema). > > class="solr.TextField" positionIncrementGap="100"> > > class="solr.WhitespaceTokenizerFactory"/> > class="solr.SynonymFilterFactory" synonyms="synonyms.txt" > ignoreCase="true" expand="false"/> > class="solr.StopFilterFactory" > > ignoreCase="true" > > words="stopwords.txt" > > enablePositionIncrements="true" > > /> > class="solr.WordDelimiterFilterFactory" > generateWordParts="1" > generateNumberParts="1" catenateWords="0" > catenateNumbers="0" > catenateAll="0" splitOnCaseChange="1"/> > class="solr.RemoveDuplicatesTokenFilterFactory"/> > > > Similar for type="query". Please advise on how to group or > cluster > document terms so that they can be used as facets. > > Many thanks in advance, > Adam Estrada >
Re: Multiple Word Facets
Use this field type - On Tue, Oct 26, 2010 at 6:43 PM, Adam Estrada wrote: > All, > I am a new to Solr faceting and stuck on how to get multiple-word > facets returned from a standard Solr query. See below for what is > currently being returned. > > > > > > 89 > 87 > 87 > 87 > 84 > 60 > 32 > 22 > 19 > 15 > 15 > 14 > 12 > 11 > 10 > 9 > 7 > 7 > 7 > 6 > 6 > 6 > 6 > ...etc... > > There are many terms in there that are 2 or 3 word phrases. For > example, Eastern Federal Lands Highway Division all gets broken down > in to the individual words that make up the total group of words. I've > seen quite a few websites that do what it is I am trying to do here so > any suggestions at this point would be great. See my schema below > (copied from the example schema). > > positionIncrementGap="100"> > > > ignoreCase="true" expand="false"/> >ignoreCase="true" >words="stopwords.txt" >enablePositionIncrements="true" >/> > generateWordParts="1" > generateNumberParts="1" catenateWords="0" catenateNumbers="0" > catenateAll="0" splitOnCaseChange="1"/> > > > > Similar for type="query". Please advise on how to group or cluster > document terms so that they can be used as facets. > > Many thanks in advance, > Adam Estrada >
Multiple Word Facets
All, I am a new to Solr faceting and stuck on how to get multiple-word facets returned from a standard Solr query. See below for what is currently being returned. 89 87 87 87 84 60 32 22 19 15 15 14 12 11 10 9 7 7 7 6 6 6 6 ...etc... There are many terms in there that are 2 or 3 word phrases. For example, Eastern Federal Lands Highway Division all gets broken down in to the individual words that make up the total group of words. I've seen quite a few websites that do what it is I am trying to do here so any suggestions at this point would be great. See my schema below (copied from the example schema). Similar for type="query". Please advise on how to group or cluster document terms so that they can be used as facets. Many thanks in advance, Adam Estrada
