indexing key value pair into lucene solr index
hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... -- -JAME
Re: indexing key value pair into lucene solr index
Hi Jame, you can - generate one token for each pair (key, value) -- key_value - insert a gap between each pair and us phrase queries - use key as field-name (if you have a restricted set of keys) - wait for joins in Solr 4.0 (http://wiki.apache.org/solr/Join) - use position or payloads to connect key and value - tell the forum your exact use-case with examples Best regrads Karsten Original-Nachricht Datum: Mon, 24 Oct 2011 17:11:49 +0530 Von: jame vaalet jamevaa...@gmail.com An: solr-user@lucene.apache.org Betreff: indexing key value pair into lucene solr index hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... -- -JAME
Re: indexing key value pair into lucene solr index
thanks karsten. can we preserve order within index field ? if yes, i can index them separately and map them using their order. On 24 October 2011 17:32, karsten-s...@gmx.de wrote: Hi Jame, you can - generate one token for each pair (key, value) -- key_value - insert a gap between each pair and us phrase queries - use key as field-name (if you have a restricted set of keys) - wait for joins in Solr 4.0 (http://wiki.apache.org/solr/Join) - use position or payloads to connect key and value - tell the forum your exact use-case with examples Best regrads Karsten Original-Nachricht Datum: Mon, 24 Oct 2011 17:11:49 +0530 Von: jame vaalet jamevaa...@gmail.com An: solr-user@lucene.apache.org Betreff: indexing key value pair into lucene solr index hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... -- -JAME -- -JAME
Re: indexing key value pair into lucene solr index
Hi Jame, preserve order in index fields: if you don't want to use phrase queries in key or value this order is position. if you use phrase queries but no value has more then 50 Tokens you also could use position and start each pair with position 100, 200, 300 ... Otherwise you could use payloads. Imho there is no standard way to connect the positions of two fields. You have to write your own Query. My Tip: Take org.apache.lucene.search.spans.TermSpans as starting point and use the queryparser-Module. btw: normaly there is a standard solution in lucene for each problem. So please tell more about your use-case and somebody will have an answer without program by your own. Best regards Karsten Original-Nachricht Datum: Mon, 24 Oct 2011 17:53:26 +0530 Von: jame vaalet jamevaa...@gmail.com An: solr-user@lucene.apache.org Betreff: Re: indexing key value pair into lucene solr index thanks karsten. can we preserve order within index field ? if yes, i can index them separately and map them using their order. On 24 October 2011 17:32, karsten-s...@gmx.de wrote: Hi Jame, you can - generate one token for each pair (key, value) -- key_value - insert a gap between each pair and us phrase queries - use key as field-name (if you have a restricted set of keys) - wait for joins in Solr 4.0 (http://wiki.apache.org/solr/Join) - use position or payloads to connect key and value - tell the forum your exact use-case with examples Best regrads Karsten Original-Nachricht Datum: Mon, 24 Oct 2011 17:11:49 +0530 Von: jame vaalet jamevaa...@gmail.com An: solr-user@lucene.apache.org Betreff: indexing key value pair into lucene solr index hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... -- -JAME -- -JAME
Re: indexing key value pair into lucene solr index
On Oct 24, 2011, at 1:41pm, jame vaalet wrote: hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... As Karsten said, providing more detail re what you're actually trying to do usually makes for better and more helpful/accurate answers. But I'm guessing you only want to search on the key, not the value, right? If so, then: 1. Create a multi-value field with a custom type, indexed, stored. 2. During indexing, add entries as keytabvalue 3. In the custom type, set the analyzer to strip off the tabvalue so you only index the key. E.g. fieldType name=key_value class=solr.TextField positionIncrementGap=100 autoGeneratePhraseQueries=true omitTermFreqAndPositions=true omitNorms=true analyzer type=index !-- Get rid of tabvalue text at the end of each string -- charFilter class=solr.PatternReplaceCharFilterFactory pattern=\t\d+$ replacement= / tokenizer class=solr.StandardTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=1 catenateNumbers=1 catenateAll=0 splitOnCaseChange=1/ /analyzer analyzer type=query tokenizer class=solr.StandardTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=0 catenateNumbers=0 catenateAll=0 splitOnCaseChange=1/ /analyzer /fieldType -- Ken -- Ken Krugler +1 530-210-6378 http://bixolabs.com custom big data solutions training Hadoop, Cascading, Mahout Solr
RE: indexing key value pair into lucene solr index
Maybe put them in a single string field (or any other field type that is not analyzed -- certainly not text) using some character separator that will connect them, but won't confuse the Solr query parser? So maybe you start out with key value pairs of Key1 value1 Key2 value2 Key3 value3 Preprocess them for indexing, and then index (and search) for them as, for example, Key1$value1 Key2$value2 Key3$value3 (You could also store their individual values in a separate field, of course). JRJ -Original Message- From: jame vaalet [mailto:jamevaa...@gmail.com] Sent: Monday, October 24, 2011 6:42 AM To: solr-user@lucene.apache.org Subject: indexing key value pair into lucene solr index hi, in my use case i have list of key value pairs in each document object, if i index them as separate index fields then in the result doc object i will get two arrays corresponding to my keys and values. The problem i face here is that there wont be any mapping between those keys and values. do we have any easy to index these data in solr ? thanks in advance ... -- -JAME
