Re: How to tell Solr to return all fields including empty fields?
One would think that a JSON output handler in Solr/Lucene would create a: "fieldname": null for missing fields. Is it possible to set that as a parameter to a json-requested output? In PHP, NULL and missing are ALMOST the exact same thing. In databases, they actually are, in my limited experience. Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Sat, 10/2/10, Khai Doan wrote: > From: Khai Doan > Subject: Re: How to tell Solr to return all fields including empty fields? > To: solr-user@lucene.apache.org > Date: Saturday, October 2, 2010, 10:36 PM > Thanks for replying. I need to > have the empty fields because if the field > is not there, the Java library ( > http://www.json.org/javadoc/org/json/JSONObject.html) > that I use to parse > the result will throw an exception. Quite > inconvenient. > > Thanks, > Khai > > On Fri, Oct 1, 2010 at 6:45 PM, Erick Erickson wrote: > > > Nope. If a field is empty, it's just not in the > document. You could index > > an empty value (some unique value for empty) if you > really need this. > > > > But a more interesting question is why you need this. > What is it about > > returning an empty field that's important? Couldn't > you detect this through > > the absence of the field? > > > > Best > > Erick > > > > On Fri, Oct 1, 2010 at 3:44 PM, Khai Doan > wrote: > > > > > Hello everyone, > > > > > > Right now, I am using fl=*,score to get all > fields from Solr. However > > > empty > > > fields are not returned. Is there a way for > me to tell Solr to return > > all > > > fields including empty fields? > > > > > > Thanks, > > > Khai > > > > > >
Re: How to tell Solr to return all fields including empty fields?
Thanks for replying. I need to have the empty fields because if the field is not there, the Java library ( http://www.json.org/javadoc/org/json/JSONObject.html) that I use to parse the result will throw an exception. Quite inconvenient. Thanks, Khai On Fri, Oct 1, 2010 at 6:45 PM, Erick Erickson wrote: > Nope. If a field is empty, it's just not in the document. You could index > an empty value (some unique value for empty) if you really need this. > > But a more interesting question is why you need this. What is it about > returning an empty field that's important? Couldn't you detect this through > the absence of the field? > > Best > Erick > > On Fri, Oct 1, 2010 at 3:44 PM, Khai Doan wrote: > > > Hello everyone, > > > > Right now, I am using fl=*,score to get all fields from Solr. However > > empty > > fields are not returned. Is there a way for me to tell Solr to return > all > > fields including empty fields? > > > > Thanks, > > Khai > > >
Re: Autosuggest with inner phrases
This's what yer lookin' for: http://www.lucidimagination.com/blog/2009/09/08/auto-suggest-from-popular-queries-using-edgengrams/ On Sat, Oct 2, 2010 at 3:14 AM, sivaprasad wrote: > > Hi , > I implemented the auto suggest using terms component.But the suggestions are > coming from the starting of the word.But i want inner phrases also.For > example, if I type "bass" Auto-Complete should offer suggestions that > include "bass fishing" or "bass guitar", and even "sea bass" (note how > "bass" is not necessarily the first word). > > How can i achieve this using solr's terms component. > > Regards, > Siva > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Autosuggest-with-inner-phrases-tp1619326p1619326.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Autosuggest with inner phrases
> Does adding the shingle filter > factory have to be done before or after indexing? It will be added to index time analyzer, if you are asking that. Also re-start solr and re-index is required.
Re: NGramFilterFactory for auto-complete that matches the middle of multi-lingual tags?
> I don't understand. Many tags like "electric吉他" or > "古典吉他" have no whitespace at all, so how does > WhitespaceTokenizer help? It makes sense for tags having more than one words. i.e. "electric guitar" If you tokenize this using whitespacetokenizer, you obtain two tokens. If you use keywordtokenizer, you obtain only one token, always. In other words, if you want query qui to return "electric guitar" you need whitespacetokenizer. analysis.jsp visualizes analysis process step by step. You can observe it.
solrj
hello,
I am trying to use solrj for interfacing with solr. I am trying to run the
SolrjTest example. I have included all the following jar files-
- commons-codec-1.3.jar
- commons-fileupload-1.2.1.jar
- commons-httpclient-3.1.jar
- commons-io-1.4.jar
- geronimo-stax-api_1.0_spec-1.0.1.jar
- apache-solr-solrj-*.jar
- wstx-asl-3.2.7.jar
- slf4j-api-1.5.5.jar
- slf4j-simple-1.5.5.jar
*My SolrjTest file is as follows:*
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrDocument;
import java.util.Map;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.SolrDocument;
import java.util.Map;
import java.util.Iterator;
import java.util.List;
import java.util.ArrayList;
import java.util.HashMap;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.client.solrj.response.FacetField;
class SolrjTest
{
public void query(String q)
{
CommonsHttpSolrServer server = null;
try
{
server = new CommonsHttpSolrServer("http://localhost:8983/solr/
");
}
catch(Exception e)
{
e.printStackTrace();
}
SolrQuery query = new SolrQuery();
query.setQuery(q);
query.setQueryType("dismax");
query.setFacet(true);
query.addFacetField("lastname");
query.addFacetField("locality4");
query.setFacetMinCount(2);
query.setIncludeScore(true);
try
{
QueryResponse qr = server.query(query);
SolrDocumentList sdl = qr.getResults();
System.out.println("Found: " + sdl.getNumFound());
System.out.println("Start: " + sdl.getStart());
System.out.println("Max Score: " + sdl.getMaxScore());
System.out.println("");
ArrayList> hitsOnPage = new
ArrayList>();
for(SolrDocument d : sdl)
{
HashMap values = new HashMap();
for(Iterator> i = d.iterator();
i.hasNext(); )
{
Map.Entry e2 = i.next();
values.put(e2.getKey(), e2.getValue());
}
hitsOnPage.add(values);
System.out.println(values.get("displayname") + " (" +
values.get("displayphone") + ")");
}
List facets = qr.getFacetFields();
for(FacetField facet : facets)
{
List facetEntries = facet.getValues();
for(FacetField.Count fcount : facetEntries)
{
System.out.println(fcount.getName() + ": " +
fcount.getCount());
}
}
}
catch (SolrServerException e)
{
e.printStackTrace();
}
}
public static void main(String[] args)
{
SolrjTest solrj = new SolrjTest();
solrj.query(args[0]);
}
}
But its giving me error as 'NoClassDefFoundError:
org/apache/solr/client/solrj/SolrServerException'.
Can anyone tell me where did i go wrong?
Re: NGramFilterFactory for auto-complete that matches the middle of multi-lingual tags?
--- On Sat, 10/2/10, Ahmet Arslan wrote: > From: Ahmet Arslan > > For example, if a user types > "guit" I want to suggest: > > "guitar" > > "electric guitar" > > "电动guitar" > > "guitar英雄" > > > > And if a user types "吉他" I want to suggest: > > "吉他Hero" > > "electric吉他" > > "古典吉他" > > > > > > I'm thinking about using: > > > > class="solr.TextField" > > positionIncrementGap="100"> > > > > > class="solr.KeywordTokenizerFactory"/> > > > class="solr.LowerCaseFilterFactory"/> > > > class="solr.NGramFilterFactory" minGramSize="1" > > maxGramSize="15" /> > > > > > > > class="solr.KeywordTokenizerFactory"/> > > > class="solr.LowerCaseFilterFactory"/> > > > > > > > > Would the above setup do what I want to do? > > fieldType autocomplete will bring you only startsWith tags > since it uses KeywordTokenizerFactory. You need > WhitespaceTokenizer for your use case. > > Or you can use two different fields and types (using > keywordtokenizer and whitespacetokenizer). So that > beginsWith matches comes first. > I don't understand. Many tags like "electric吉他" or "古典吉他" have no whitespace at all, so how does WhitespaceTokenizer help?
timestamp column
Is there a timestamp column in Solr,i.e. I could feed it something like: 2010-10-15T23:59:59 And it's indexable of course :-) Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php
Re: Autosuggest with inner phrases
Does adding the shingle filter factory have to be done before or after indexing? Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Sat, 10/2/10, Ahmet Arslan wrote: > From: Ahmet Arslan > Subject: Re: Autosuggest with inner phrases > To: solr-user@lucene.apache.org > Date: Saturday, October 2, 2010, 5:10 AM > > Hi , > > I implemented the auto suggest using terms > component.But > > the suggestions are > > coming from the starting of the word.But i want inner > > phrases also.For > > example, if I type "bass" Auto-Complete should offer > > suggestions that > > include "bass fishing" or "bass guitar", and even > > "sea bass" (note how > > "bass" is not necessarily the first word). > > > > How can i achieve this using solr's terms component. > > > You can add ShingleFilterFactory to your field type > definition. > > http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ShingleFilterFactory > > > >
Re: NGramFilterFactory for auto-complete that matches the middle of multi-lingual tags?
> For example, if a user types "guit" I want to suggest: > "guitar" > "electric guitar" > "电动guitar" > "guitar英雄" > > And if a user types "吉他" I want to suggest: > "吉他Hero" > "electric吉他" > "古典吉他" > > > I'm thinking about using: > > positionIncrementGap="100"> > > class="solr.KeywordTokenizerFactory"/> > class="solr.LowerCaseFilterFactory"/> > class="solr.NGramFilterFactory" minGramSize="1" > maxGramSize="15" /> > > > class="solr.KeywordTokenizerFactory"/> > class="solr.LowerCaseFilterFactory"/> > > > > Would the above setup do what I want to do? fieldType autocomplete will bring you only startsWith tags since it uses KeywordTokenizerFactory. You need WhitespaceTokenizer for your use case. Or you can use two different fields and types (using keywordtokenizer and whitespacetokenizer). So that beginsWith matches comes first.
Re: Upgrade to Solr 1.4, very slow at start up when loading all cores
On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun wrote: > Hi Yonik, > > I attached the solrconfig.xml to you in previous post, and we do have > firstSearch and newSearch hook ups. > > I commented them out, all 130 cores loaded up in 1 minute, same as in solr > 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took > about 6.5GB for same amount of data. For other's reference: here is the warming query (it's the same for newSearcher too): type:message 0 10 message_date desc The sort field message_date is what will be taking up the memory. Starting with Lucene 2.9 (which is used in Solr 1.4), searching and sorting is per-segment. This is generally beneficial, but in this case I believe it is causing the extra memory usage because the same date value that would have been shared across all documents in the fieldcache is now repeated in each segment it is used in. One potential fix (that requires you to reindex) is to use the "date" fieldType as defined in the new 1.4 schema: This will use 8 bytes per document in your index, rather than 4 bytes per doc + an array of unique string-date values per index. Trunk (4.0-dev) is also much more efficient at storing string-based fields in the FieldCache - but that will only help you if you're comfortable with using development versions. -Yonik http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8
Re: Search Interface
tks guys!! working now.. =) On Tue, Sep 28, 2010 at 5:39 PM, Lance Norskog wrote: > There is already a simple Velocity app. Just hit > http://localhost:8983/solr/browse. > You can configure some handy parameters to make walkable facets in > solrconfig.xml. > > On Tue, Sep 28, 2010 at 5:23 AM, Antonio Calo' > wrote: > > Hi > > > > You could try to use the Velocity framework to build GUIs in a quick and > > efficent manner. > > > > Solr come with a velocity handler already integrated that could be the > best > > solution in your case: > > > > http://wiki.apache.org/solr/VelocityResponseWriter > > > > Also take these hints on the same topic: > > > http://www.lucidimagination.com/blog/2009/11/04/solritas-solr-1-4s-hidden-gem/ > > > > there is also a webinar about rapid prototyping with solr: > > > > > http://www.slideshare.net/erikhatcher/rapid-prototyping-with-solr-4312681 > > > > Hope this help > > > > Antonio > > > > > > Il 28/09/2010 4.35, Claudio Devecchi ha scritto: > >> > >> Hi everybody, > >> > >> I`m implementing my first solr engine for conceptual tests, I`m crawling > >> my > >> wiki intranet to make some searches, the engine is working fine already, > >> but > >> I need some interface to make my searchs. > >> Somebody knows where can I find some search interface just for > >> customizations? > >> > >> Tks > > > > > > > > -- > Lance Norskog > goks...@gmail.com > -- Claudio Devecchi flickr.com/cdevecchi
Re: Autosuggest with inner phrases
> Hi , > I implemented the auto suggest using terms component.But > the suggestions are > coming from the starting of the word.But i want inner > phrases also.For > example, if I type "bass" Auto-Complete should offer > suggestions that > include "bass fishing" or "bass guitar", and even > "sea bass" (note how > "bass" is not necessarily the first word). > > How can i achieve this using solr's terms component. You can add ShingleFilterFactory to your field type definition. http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ShingleFilterFactory
Autosuggest with inner phrases
Hi , I implemented the auto suggest using terms component.But the suggestions are coming from the starting of the word.But i want inner phrases also.For example, if I type "bass" Auto-Complete should offer suggestions that include "bass fishing" or "bass guitar", and even "sea bass" (note how "bass" is not necessarily the first word). How can i achieve this using solr's terms component. Regards, Siva -- View this message in context: http://lucene.472066.n3.nabble.com/Autosuggest-with-inner-phrases-tp1619326p1619326.html Sent from the Solr - User mailing list archive at Nabble.com.
NGramFilterFactory for auto-complete that matches the middle of multi-lingual tags?
I working on a user-generated tagging feature. Some of the tags could be multi-lingual, mixng languages like English, Chinese, Japanese I'd like to add auto-complete to help users to enter the tags. And I'd want to match in the middle of the tags as well. For example, if a user types "guit" I want to suggest: "guitar" "electric guitar" "电动guitar" "guitar英雄" And if a user types "吉他" I want to suggest: "吉他Hero" "electric吉他" "古典吉他" I'm thinking about using: Would the above setup do what I want to do? Also how would I deal with hyphens? For example I want an input or either "wi-f" or "wif" to match the tag "wi-fi". Would adding WordDelimiterFilterFactory to both "index" and "query" accomplish that? Thanks.
