Thanks Otis & Mike.
Probably we should keep it the way it is now. Would be better to include
more information on the various combinations of these options and its effect
on the final result (set of terms that get to the index). Would be nicer if
we could mention the search scenario as well. To be honest, it took me a
while to get a grip on it.
On the same topic, what would be the effect of the following code.
Document doc = new Document();
Field f = new Field("body", bodyText, Field.Store.NO
,Field.Index.TOKENIZED);
f.setOmitNorms(true);
Would that be equivalent to
Document doc = new Document();
Field f = new Field("body", bodyText, Field.Store.NO ,Field.Index.NO_NORMS);
And Field.Index.TOKENIZED has no effect after f.setOmitNorms(true); ?
Many thanks,
Dino
-----Original Message-----
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 27 August 2008 10:37
To: java-user@lucene.apache.org
Subject: Re: Case Sensitivity
Actually, as confusing as it is, Field.Index.NO_NORMS means
Field.Index.UN_TOKENIZED plus field.setOmitNorms(true).
Probably we should rename it to Field.Index.UN_TOKENiZED_NO_NORMS?
Mike
Otis Gospodnetic wrote:
> Dino, you lost me half-way through your email :(
>
> NO_NORMS does not mean the field is not tokenized.
> UN_TOKENIZED does mean the field is not tokenized.
>
>
> Otis--
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Dino Korah <[EMAIL PROTECTED]>
>> To: java-user@lucene.apache.org
>> Sent: Tuesday, August 26, 2008 9:17:49 AM
>> Subject: RE: Case Sensitivity
>>
>> I think I should rephrase my question.
>>
>> [ Context: Using out of the box StandardAnalyzer for indexing and
>> searching.
>> ]
>>
>> Is it right to say that a field, if either UN_TOKENIZED or NO_NORMS-
>> ized (
>> field.setOmitNorms(true) ), it doesn't get analyzed while indexing?
>> Which means that when we search, it gets thru the analyzer and we
>> need to analyze them differently in the analyzer we use for
>> searching?
>> Doesn't it mean that a setOmitNorms(true) field also doesn't get
>> tokenized?
>>
>> What is the best solution if one was to add a set of fields
>> UN_TOKENIZED and others TOKENIZED, of the later set a few with
>> setOmitNorms(true) (the index writer is plain StandardAnalyzer
>> based)? A per field analyzer at query time ?!
>>
>> Many thanks,
>> Dino
>>
>>
>> -----Original Message-----
>> From: Dino Korah [mailto:[EMAIL PROTECTED]
>> Sent: 26 August 2008 12:12
>> To: 'java-user@lucene.apache.org'
>> Subject: RE: Case Sensitivity
>>
>> A little more case sensitivity questions.
>>
>> Based on the discussion on
>> http://markmail.org/message/q7dqr4r7o6t6dgo5
>> and
>> on this thread, is it right to say that a field, if either
>> UN_TOKENIZED or NO_NORMS-ized, it doesn't get analyzed while
>> indexing? Which means we need to case-normalize (down-case) those
>> fields before hand?
>>
>> Doest it mean that if I can afford, I should use norms.
>>
>> Many thanks,
>> Dino
>>
>>
>>
>> -----Original Message-----
>> From: Steven A Rowe [mailto:[EMAIL PROTECTED]
>> Sent: 19 August 2008 17:43
>> To: java-user@lucene.apache.org
>> Subject: RE: Case Sensitivity
>>
>> Hi Dino,
>>
>> I think you'd benefit from reading some FAQ answers, like:
>>
>> "Why is it important to use the same analyzer type during indexing
>> and search?"
>>
>> 4472d10961ba63c>
>>
>> Also, have a look at the AnalysisParalysis wiki page for some hints:
>>
>>
>> On 08/19/2008 at 8:57 AM, Dino Korah wrote:
>>> From the discussion here what I could understand was, if I am using
>>> StandardAnalyzer on TOKENIZED fields, for both Indexing and
>>> Querying, I shouldn't have any problems with cases.
>>
>> If by "shouldn't have problems with cases" you mean "can match
>> case-insensitively", then this is true.
>>
>>> But if I have any UN_TOKENIZED fields there will be problems if I do
>>> not case-normalize them myself before adding them as a field to the
>>> document.
>>
>> Again, assuming that by "case-normalize" you mean "downcase", and
>> that you want case-insensitive matching, and that you use the
>> StandardAnalyzer (or some other downcasing analyzer) at query-time,
>> then this is true.
>>
>>> In my case I have a mixed scenario. I am indexing emails and the
>>> email addresses are indexed UN_TOKENIZED. I do have a second set of
>>> custom tokenized field, which keep the tokens in individual fields
>>> with same name.
>> [...]
>>> Does it mean that where ever I use UN_TOKENIZED, they do not get
>>> through the StandardAnalyzer before getting Indexed, but they do
>>> when they are searched on?
>>
>> This is true.
>>
>>> If that is the case, Do I need to normalise them before adding to
>>> document?
>>
>> If you want case-insensitive matching, then yes, you do need to
>> normalize them before adding them to the document.
>>
>>> I also would like to know if it is better to employ an EmailAnalyzer
>>> that makes a TokenStream out of the given email address, rather than
>>> using a simplistic function that gives me a list of string pieces
>>> and adding them one by one. With searches, would both the approaches
>>> give same result?
>>
>> Yes, both approaches give the same result. When you add string
>> pieces one-by-one, you are adding multiple same-named fields. By
>> contrast, the EmailAnalyzer approach would add a single field, and
>> would allow you to control positions (via
>> Token.setPositionIncrement():
>>
>> ml#setPositionIncrement(int)>), e.g. to improve phrase handling.
>> Also, if
>> you make up an EmailAnalyzer, you can use it to search against your
>> tokenized email field, along with other analyzer(s) on other
>> field(s), using the PerFieldAnalyzerWrapper
>>
>> AnalyzerWrapper.html>.
>>
>> Steve
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
