Design Question - Dynamic Field Names (*)

[Kumaravel Kandasami]

Need to your suggestion in  best designing the following requirement.

- We have two indexes.
Index 1: "name_index",
Fields:
"id" - indexed, not stored
 "field_name" - indexed, stored.

Index 2: "trans_index',
Fields(Dynamic Schema):
"id" - indexed, not stored
"*" - indexed, stored.

(Dynamic field names of the trans_index is the same as the "field_name" from
the name_index.)

- Requirement:

User would select the field he wants to query from the "name_index".
Once he selects the one of the values from the 'field_name' (from the
name_index), he queries the trans_index using the field_name.


- Issue:

When indexing the name_index field:"field_name" we are using the analyzer
that would lowercase, strip spaces etc.
Example: "First Name", "firstName" values are all stored and indexed as
'firstname'.

However, when we store field names in the trans_index we would be storing as
it is ... without analyzing.
So User queries like 'firstname:a*' might not match.

- Possible Solution:

We are planning to have an custom analyzer that we would use while indexing
(configured in the schema.xml) file. As well the crawler program would use
the
same analyzer to create field names.

Is there any better design solutions ?


Kumar_/|\_
www.saisk.com
ku...@saisk.com
"making a profound difference with knowledge and creativity..."

copyField to a dynamic field

[Paul Terray]

Hi,

 

I know this is a complex one, but it help me to be able to make a dynamic
copy field, like:

   

 

The goal is to have for each string index a tokenized one.

 

It does not seem possible at the moment, but will it be in the foreseeable
future?

 

Thanks anyway !

 


> 

Paul Terray 


  

Consultant Avant-Vente


> 

SOLLAN

 


  

27, bis rue du Progrès 
93100 Montreuil - France
Tel :  +33 (0)1 48 51 15 44
Fax : +33 (0)1 48 51 15 48
  [EMAIL PROTECTED]
  www.sollan.com

STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and
proprietary material for the sole use of the intended recipient. Any review
or distribution by others is strictly prohibited. If you are not the
intended recipient please contact the sender and delete all copies. 


  SOLLAN

 

strange search-behavior over dynamic field

[stockii]

Hello.

i have the field reason_1 and reason_2. this two fields is in my schema one
dynamicField: 

i copy this field in my text-default search field: 
And in a new field reason: 

---> if i have two documents with the exactly same value in the reason_1
field, solr can only find ONE document, not both.

why ? is it a behavior of solr or a wrong usage of me ? 

-
--- System


One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 
1 Core with 31 Million Documents other Cores < 100.000

- Solr1 for Search-Requests - commit every Minute  - 4GB Xmx
- Solr2 for Update-Request  - delta every 2 Minutes - 4GB Xmx
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/strange-search-behavior-over-dynamic-field-tp2508711p2508711.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Design Question - Dynamic Field Names (*)

[Shalin Shekhar Mangar]

On Sat, Jan 16, 2010 at 3:33 AM, Kumaravel Kandasami <
kumaravel.kandas...@gmail.com> wrote:

> Need to your suggestion in  best designing the following requirement.
>
> - We have two indexes.
> Index 1: "name_index",
> Fields:
> "id" - indexed, not stored
>  "field_name" - indexed, stored.
>
> Index 2: "trans_index',
> Fields(Dynamic Schema):
> "id" - indexed, not stored
> "*" - indexed, stored.
>
> (Dynamic field names of the trans_index is the same as the "field_name"
> from
> the name_index.)
>
> - Requirement:
>
> User would select the field he wants to query from the "name_index".
> Once he selects the one of the values from the 'field_name' (from the
> name_index), he queries the trans_index using the field_name.
>
>
> - Issue:
>
> When indexing the name_index field:"field_name" we are using the analyzer
> that would lowercase, strip spaces etc.
> Example: "First Name", "firstName" values are all stored and indexed as
> 'firstname'.
>
> However, when we store field names in the trans_index we would be storing
> as
> it is ... without analyzing.
> So User queries like 'firstname:a*' might not match.
>
> - Possible Solution:
>
> We are planning to have an custom analyzer that we would use while indexing
> (configured in the schema.xml) file. As well the crawler program would use
> the
> same analyzer to create field names.
>
> Is there any better design solutions ?
>
>
Your scenario sounds quite strange and it is still not clear why you are
doing all this. Perhaps the solution doesn't even require two indexes? Can
you describe the actual problem so that we can be of more help?

-- 
Regards,
Shalin Shekhar Mangar.

Re: Design Question - Dynamic Field Names (*)

[Kumaravel Kandasami]

First Thanks for the response.

Yes, mostly likely we want to optimize to one index file. I think it is
possible, coming from the RDBMS world  we might be over complicating the
solution.

*Requirement:*
- We are indexing CSV files and generating field names dynamically from the
"header" line.
User should be able to *list all the possible header names* (i.e. dynamic
field names), and filter results based on some of the field names.

- Also, list* all possible values* associated to for a given field name.




Kumar_/|\_
www.saisk.com
ku...@saisk.com
"making a profound difference with knowledge and creativity..."


On Tue, Jan 19, 2010 at 1:33 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> On Sat, Jan 16, 2010 at 3:33 AM, Kumaravel Kandasami <
> kumaravel.kandas...@gmail.com> wrote:
>
> > Need to your suggestion in  best designing the following requirement.
> >
> > - We have two indexes.
> > Index 1: "name_index",
> > Fields:
> > "id" - indexed, not stored
> >  "field_name" - indexed, stored.
> >
> > Index 2: "trans_index',
> > Fields(Dynamic Schema):
> > "id" - indexed, not stored
> > "*" - indexed, stored.
> >
> > (Dynamic field names of the trans_index is the same as the "field_name"
> > from
> > the name_index.)
> >
> > - Requirement:
> >
> > User would select the field he wants to query from the "name_index".
> > Once he selects the one of the values from the 'field_name' (from the
> > name_index), he queries the trans_index using the field_name.
> >
> >
> > - Issue:
> >
> > When indexing the name_index field:"field_name" we are using the analyzer
> > that would lowercase, strip spaces etc.
> > Example: "First Name", "firstName" values are all stored and indexed as
> > 'firstname'.
> >
> > However, when we store field names in the trans_index we would be storing
> > as
> > it is ... without analyzing.
> > So User queries like 'firstname:a*' might not match.
> >
> > - Possible Solution:
> >
> > We are planning to have an custom analyzer that we would use while
> indexing
> > (configured in the schema.xml) file. As well the crawler program would
> use
> > the
> > same analyzer to create field names.
> >
> > Is there any better design solutions ?
> >
> >
> Your scenario sounds quite strange and it is still not clear why you are
> doing all this. Perhaps the solution doesn't even require two indexes? Can
> you describe the actual problem so that we can be of more help?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Design Question - Dynamic Field Names (*)

[Chris Hostetter]

: - We are indexing CSV files and generating field names dynamically from the
: "header" line.
: User should be able to *list all the possible header names* (i.e. dynamic
: field names), and filter results based on some of the field names.
: - Also, list* all possible values* associated to for a given field name.

#1) the LukeRequestHandler can list all field names in the index.
#2) the TermsComponent or Faceting can list all *indexed* values in a 
given field ... which one you'll want to use depends largely on what you 
want to do with that list.



-Hoss

Re: copyField to a dynamic field

[Chris Hostetter]

: I know this is a complex one, but it help me to be able to make a dynamic
: copy field, like:
:
:

: The goal is to have for each string index a tokenized one.

: It does not seem possible at the moment, but will it be in the foreseeable
: future?

I assume you mean you are interested in supporting something like that
where whatever the "*" matches in the "source" is also matches in the
"target" right? ... I don't know ofanyone working on it right now, but
it's certainly something i can imagine might be implemented at some point
in the future .. it just needs someoen with enough interest in submitting
a patch :)



-Hoss

Re: copyField to a dynamic field

[Mike Klaas]

On 9/20/06, Chris Hostetter <[EMAIL PROTECTED]> wrote:

:



I assume you mean you are interested in supporting something like that
where whatever the "*" matches in the "source" is also matches in the
"target" right? ... I don't know ofanyone working on it right now, but
it's certainly something i can imagine might be implemented at some point
in the future .. it just needs someoen with enough interest in submitting
a patch :)


It would probably be easier to support an arbitrary regexp:


or


-Mike

listing dynamic field in search results

[Kevin Osborn]

I have quite a few dynamic fields. However, I will usually only want to return 
just a couple of those fields.

So, if my static fields are StaticField1 and StaticField2 and my dynamic fields 
are s_DynamicField1, sf_DynamicField2, etc., the following line will work as 
expected:

http://localhost:8080/q=...&fl=StaticField1,StaticField2,*

This will return all 4 fields.

But http://localhost:8080/q=...&fl=StaticField1,StaticField2,s_DynamicField1 
will only return StaticField1 and StaticField2.

Is this a bug? Expected behavior?

My dynamic fields are string or sfloat. They are index, stored, and 
multivalued. I am also using a pretty new build of Solr.

Re: strange search-behavior over dynamic field

[Erick Erickson]

What does the admin page show you are the contents of
your index for reason_1?

I suspect you don't really have two documents with the same
value. Perhaps you give them both the same uniqueKey and
one overwrites the other. Perhaps you didn't commit the second.
Perhaps

But you haven't provided enough information to go on here. Where
is the query (don't forget &debugQuery=on). Where is the input?

Best
Erick

On Wed, Feb 16, 2011 at 6:26 AM, stockii  wrote:
>
> Hello.
>
> i have the field reason_1 and reason_2. this two fields is in my schema one
> dynamicField:  stored="false"/>
>
> i copy this field in my text-default search field:  source="reason_*" dest="text"/>
> And in a new field reason: 
>
> ---> if i have two documents with the exactly same value in the reason_1
> field, solr can only find ONE document, not both.
>
> why ? is it a behavior of solr or a wrong usage of me ?
>
> -
> --- System
> 
>
> One Server, 12 GB RAM, 2 Solr Instances, 7 Cores,
> 1 Core with 31 Million Documents other Cores < 100.000
>
> - Solr1 for Search-Requests - commit every Minute  - 4GB Xmx
> - Solr2 for Update-Request  - delta every 2 Minutes - 4GB Xmx
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/strange-search-behavior-over-dynamic-field-tp2508711p2508711.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: strange search-behavior over dynamic field

[stockii]

the fieldType is textgen. 

-
--- System


One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 
1 Core with 31 Million Documents other Cores < 100.000

- Solr1 for Search-Requests - commit every Minute  - 4GB Xmx
- Solr2 for Update-Request  - delta every 2 Minutes - 4GB Xmx
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/strange-search-behavior-over-dynamic-field-tp2508711p2509166.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: strange search-behavior over dynamic field

[stockii]

the documents havent the same uniquekey, only reason is the same.

i cannot show the exactly search request, because of privacy policy... 
the query is like that: 
reason_1: firstname lastname, 
reason_2: 1234, 02.02.2011
--> in field reason: firstname lastname, 1234, 02.02.2011

the search request is form an PHP-Application. On my TestEnvironment i
cannot rebuild this case ... =((



okay ... i dont know why, but after a delta-import, its all okay ...

-
--- System


One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 
1 Core with 31 Million Documents other Cores < 100.000

- Solr1 for Search-Requests - commit every Minute  - 4GB Xmx
- Solr2 for Update-Request  - delta every 2 Minutes - 4GB Xmx
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/strange-search-behavior-over-dynamic-field-tp2508711p2509610.html
Sent from the Solr - User mailing list archive at Nabble.com.

Saving dynamic field name without dynamic extension

[Debra]

I want to add a suffix to my fields names to use the dynamic fields feature.
Is there a way to save the field name without the suffix so users can search
by field  with plain field name?

-- 
View this message in context: 
http://www.nabble.com/Saving-dynamic-field-name-without-dynamic-extension-tf3358269.html#a9340901
Sent from the Solr - User mailing list archive at Nabble.com.

Re: listing dynamic field in search results

[Yonik Seeley]

On 4/11/07, Kevin Osborn <[EMAIL PROTECTED]> wrote:

I have quite a few dynamic fields. However, I will usually only want to return 
just a couple of those fields.

So, if my static fields are StaticField1 and StaticField2 and my dynamic fields 
are s_DynamicField1, sf_DynamicField2, etc., the following line will work as 
expected:

http://localhost:8080/q=...&fl=StaticField1,StaticField2,*

This will return all 4 fields.

But http://localhost:8080/q=...&fl=StaticField1,StaticField2,s_DynamicField1 
will only return StaticField1 and StaticField2.

Is this a bug? Expected behavior?


That would be a bug... but I can't reproduce it.

With the example solr server, loaded with the example data, the
following query returns both fields:
http://localhost:8983/solr/select/?q=solr&fl=name,incubationdate_dt

[...]


  2006-01-17T00:00:00.000Z
  Solr, the Enterprise Search Server




Note that incubation_dt is a dynamic field.

-Yonik

Re: listing dynamic field in search results

[Chris Hostetter]

: But
: http://localhost:8080/q=...&fl=StaticField1,StaticField2,s_DynamicField1
: will only return StaticField1 and StaticField2.

that doesn't sound right.   If i use the example configs/docs and hit
this query...

http://localhost:8983/solr/select?q=solr&&fl=name%2Cincubationdate_dt

...i get both fields, and incubationdate_dt is a dynamic field.


-Hoss

Re: listing dynamic field in search results

[Kevin Osborn]

I discovered the issue. Some of my dynamic field names have spaces in them. If 
I replace ' ' with '_', then it works fine. This really isn't a great solution, 
since a '_' could be a legitimate part of the field name. I tried enclosing the 
entire field name with quotes, so the query is:

http://localhost:8080/q=...&fl=StaticField1,StaticField2,%22s_Dynamic+Field+1%22

I am guessing that Solr is parsing "s_Dynamic", "Field", and "1" as separate 
fields. Does anyone know a way around this? My initial testing shows that I 
have a similar problem if I query by that field as well. I was hoping that Solr 
would interpret text enclosed by quotes as a single field, but this does not 
appear to be the case.

Also, this all works fine in a facet count for facet.field. But in that case, 
the parameter take a single field name.

facet.query also fails. So, perhaps this is a Lucene issue?

- Original Message 
From: Chris Hostetter <[EMAIL PROTECTED]>
To: Solr <[EMAIL PROTECTED]>
Sent: Wednesday, April 11, 2007 5:30:57 PM
Subject: Re: listing dynamic field in search results


: But
: http://localhost:8080/q=...&fl=StaticField1,StaticField2,s_DynamicField1
: will only return StaticField1 and StaticField2.

that doesn't sound right.   If i use the example configs/docs and hit
this query...

http://localhost:8983/solr/select?q=solr&&fl=name%2Cincubationdate_dt

...i get both fields, and incubationdate_dt is a dynamic field.


-Hoss

Re: listing dynamic field in search results

[Yonik Seeley]

On 4/12/07, Kevin Osborn <[EMAIL PROTECTED]> wrote:

I discovered the issue. Some of my dynamic field names have spaces in them. If 
I replace ' ' with '_', then it works fine. This really isn't a great solution, 
since a '_' could be a legitimate part of the field name.


Do you not have any control over the base names of your dynamic fields?


facet.query also fails. So, perhaps this is a Lucene issue?


Most likely... I doubt the Lucene QueryParser can parse field names with spaces.

-Yonik

Re: listing dynamic field in search results

[Kevin Osborn]

The field names come from another source, most of which have spaces. So, it 
looks like I will have to do a conversion between the space and some other 
uncommon character.

Does anyone know the range of valid characters for field names in the Lucene 
QueryParser? Obviously spaces are out of the range. I couldn't find this 
information in the Lucene documentation.

- Original Message 
From: Yonik Seeley <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thursday, April 12, 2007 9:56:18 AM
Subject: Re: listing dynamic field in search results

On 4/12/07, Kevin Osborn <[EMAIL PROTECTED]> wrote:
> I discovered the issue. Some of my dynamic field names have spaces in them. 
> If I replace ' ' with '_', then it works fine. This really isn't a great 
> solution, since a '_' could be a legitimate part of the field name.

Do you not have any control over the base names of your dynamic fields?

> facet.query also fails. So, perhaps this is a Lucene issue?

Most likely... I doubt the Lucene QueryParser can parse field names with spaces.

-Yonik

Re: listing dynamic field in search results

[Chris Hostetter]

: 
http://localhost:8080/q=...&fl=StaticField1,StaticField2,%22s_Dynamic+Field+1%22
:
: I am guessing that Solr is parsing "s_Dynamic", "Field", and "1" as
: separate fields. Does anyone know a way around this? My initial testing

This is because the fl param does some pattern splitting to figure out the
names of the fields.  space "," and "|" are all considered delimiters in
the "fl" param ... it goes back to when Solr didn't have multi-value
params ... now that it does, we could say that specifying multiple fl
params means that each param is treated as an exact field name, but i
think in the common case people would still wnat some splitting to happen
... consider the case of having a "default" fl as well as an "appends"
list containing score, so you ensure people allways get it -- but you
still want them to be able to type fl=a,b,c instead of fl=a&fl=b&fl=c.

if someone wants to submit a patch to make fl multivalue, and add a new
config block to solrconfig.xml with info on parsing field names from
params, i would totally be on board that ... in addition to a
"spiltPattern" (which could be unspecificed being no spliting) this would
be a good place to specify what field alias to use for the "score" (if you
want to call it something else) and what field alias people should use if
they want raw index order when sorting (something else that was brought up
recently)

: shows that I have a similar problem if I query by that field as well. I
: was hoping that Solr would interpret text enclosed by quotes as a single
: field, but this does not appear to be the case.

: facet.query also fails. So, perhaps this is a Lucene issue?

on the query side this would definiltey be a QueryParser issue ... i'm not
sure what exactly QueryParser consideres legal field name characters, but
it certianly doesn't include spaces ... you've have to heck the
QueryParser.jj file.

(I think it's sae to say that the Lucene Java community wouldn't turn down
a patch to the querysyntax doc clarifying what rules QueryParser enforces
for field names)




-Hoss

Proposition of a new feature: Dynamic Field Types

[nicolas . dessaigne]

Dynamic field types are field types that act as proxies to other field
types. The choice of the field type to use is done on a per document basis
and is dependent of the values of the document's fields.

The use case that led us to this feature is the indexation of documents in
different languages. We use a specific analyzer for each language but want
to index semantic information that is not specific to the language.

For example, we would add in the index the semantic tag {co:Paris} for the
expressions "Paris", "capital city of France", "the city of lights" in
English and "Paris", "capitale de la France", "la ville lumière" in French.
This allows us to provide advanced functionalities such as semantic and
cross-lingual search.

To do so in SOLR, we chose to index texts written in different languages in
the same field, while analyzing them with different analyzers. Hence the
proposition of a new feature that respond to this need: Dynamic Field Types.

The idea of this new field type is to act as a proxy to other field types.
Depending of the values of some fields of the document to index, it chooses
the correct field type to use. In our situation, we use it to choose the
correct language dependent field type based on the value of the field named
"language". It is configured with a config similar to the following:


...



...










The last condition is used as a catch-all if preceding conditions are not
met.

What do you think of this feature?

Best regards,
Nicolas Dessaigne

Re: Saving dynamic field name without dynamic extension

[Mike Klaas]

On 3/6/07, Debra <[EMAIL PROTECTED]> wrote:


I want to add a suffix to my fields names to use the dynamic fields feature.
Is there a way to save the field name without the suffix so users can search
by field  with plain field name?


No, and I'm not sure that it is possible.  Solr needs to know the type
of a field at all times--not just during indexing.

Why not create a _user suffix, and programmatically add the suffix to
user queries before it reaches solr?

-Mike

How to Create dynamic field names using script transformers

[JavaGuy84]

Hi,

I am trying to generate a dynamic fieldname using custom transformers but
couldn't achieve the expected results.

My requirement is that I do not want to hardcode some of field names used by
SOLR for indexing, instead the field name should be generated using the data
retreieved from a table.

Any help on this regard is greatly appreciated.

Thanks,
Barani
-- 
View this message in context: 
http://old.nabble.com/How-to-Create-dynamic-field-names-using-script-transformers-tp27329876p27329876.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Proposition of a new feature: Dynamic Field Types

[Grant Ingersoll]

Why can't you choose the proper field in your application and keep  
separate fields per language?  Putting them all in the same field,  
regardless of language, is not a good idea in my opinion because it is  
more than likely going to skew your statistics and lower your relevance.


That being said, the dynamic field type is still an interesting idea.

-Grant

On Feb 29, 2008, at 5:56 AM, [EMAIL PROTECTED] wrote:


Dynamic field types are field types that act as proxies to other field
types. The choice of the field type to use is done on a per document  
basis

and is dependent of the values of the document's fields.

The use case that led us to this feature is the indexation of  
documents in
different languages. We use a specific analyzer for each language  
but want

to index semantic information that is not specific to the language.

For example, we would add in the index the semantic tag {co:Paris}  
for the

expressions "Paris", "capital city of France", "the city of lights" in
English and "Paris", "capitale de la France", "la ville lumière" in  
French.
This allows us to provide advanced functionalities such as semantic  
and

cross-lingual search.

To do so in SOLR, we chose to index texts written in different  
languages in
the same field, while analyzing them with different analyzers. Hence  
the
proposition of a new feature that respond to this need: Dynamic  
Field Types.


The idea of this new field type is to act as a proxy to other field  
types.
Depending of the values of some fields of the document to index, it  
chooses
the correct field type to use. In our situation, we use it to choose  
the
correct language dependent field type based on the value of the  
field named

"language". It is configured with a config similar to the following:


...



...










The last condition is used as a catch-all if preceding conditions  
are not

met.

What do you think of this feature?

Best regards,
Nicolas Dessaigne

RE: Proposition of a new feature: Dynamic Field Types

[nicolas . dessaigne]

Thanks for your response Grant.

You are right, depending of the language we could index the text in a
specific field. At request time, we would then ask all the fields for the
query.

I see however a few possible problems with this approach. By order of
decreasing importance:

- Influence on relevance

I assume the idf is calculated on a field by field basis? In the context of
one field per language, the documents whose language is the less present in
the index will receive an unusual boost for cross-lingual tokens. This
situation can be quite frequent as the distribution of languages in the
index is usually heterogeneous. Even if it was homogeneous, we would have
the problem with rare text in one language citing words in another.

On the other hand, you are right in the sense that the idf of language
specific words is also altered. In the context of one field for all
languages, the idf could be very low for a word if it is a common word in
another language. For example, the world "thé" in French is quite rare, but
its idf would be greatly altered by the word "the" in English.

We have a dilemma here...

- Performance

Queries are in O(log n) if I'm not mistaken? Then a disjunction query on x
language fields would be nearly x times slower, no?

- Verbose configuration

Not an important point, but with the dynamic field type, you configure only
one time all the languages. Otherwise, you must do so for each text field.

The query handler configuration would also be much more verbose. We usually
use the dismax handler and the qf could become very long.

- Highlight

Not an important point either, but a bit of work need to be done to
aggregate the results.

In conclusion, the choice is not so clear for me. Your remark on the
relevance made me think a bit more on multilingual problems. There may be a
way to tune the idf of some fields depending on others?

Another idea would be to boost documents in the language of the request.
This may be actually much simpler.

If you have any idea on the subject I'm very interested!

Nicolas


-Message d'origine-
De : Grant Ingersoll [mailto:[EMAIL PROTECTED] 
Envoyé : vendredi 29 février 2008 14:06
À : solr-user@lucene.apache.org
Objet : Re: Proposition of a new feature: Dynamic Field Types

Why can't you choose the proper field in your application and keep  
separate fields per language?  Putting them all in the same field,  
regardless of language, is not a good idea in my opinion because it is  
more than likely going to skew your statistics and lower your relevance.

That being said, the dynamic field type is still an interesting idea.

-Grant

On Feb 29, 2008, at 5:56 AM, [EMAIL PROTECTED] wrote:

> Dynamic field types are field types that act as proxies to other field
> types. The choice of the field type to use is done on a per document  
> basis
> and is dependent of the values of the document's fields.
>
> The use case that led us to this feature is the indexation of  
> documents in
> different languages. We use a specific analyzer for each language  
> but want
> to index semantic information that is not specific to the language.
>
> For example, we would add in the index the semantic tag {co:Paris}  
> for the
> expressions "Paris", "capital city of France", "the city of lights" in
> English and "Paris", "capitale de la France", "la ville lumière" in  
> French.
> This allows us to provide advanced functionalities such as semantic  
> and
> cross-lingual search.
>
> To do so in SOLR, we chose to index texts written in different  
> languages in
> the same field, while analyzing them with different analyzers. Hence  
> the
> proposition of a new feature that respond to this need: Dynamic  
> Field Types.
>
> The idea of this new field type is to act as a proxy to other field  
> types.
> Depending of the values of some fields of the document to index, it  
> chooses
> the correct field type to use. In our situation, we use it to choose  
> the
> correct language dependent field type based on the value of the  
> field named
> "language". It is configured with a config similar to the following:
>
>   
>   ...
>   
>
>   
>   ...
>   
>
>   
>   
>name="french_ft"/>
>name="english_ft"/>
>   
>   
>   
>
> The last condition is used as a catch-all if preceding conditions  
> are not
> met.
>
> What do you think of this feature?
>
> Best regards,
> Nicolas Dessaigne

Re: Proposition of a new feature: Dynamic Field Types

[Grant Ingersoll]

How many languages are you dealing with?  How are you generating your  
queries?  Are you taking your source language and translating it to  
each of the languages?  Or are you just targeting one source to  
destination language?  Back when I was doing CLIR, I would create  
separate indexes, but that is not to say it is appropriate for your  
task.  It was always highly suspect in my mind the notion of combining  
the results of multiple languages into a single hit list.  I just  
doubt that the scores are meaningful in that way since scores should  
generally only be considered relative to each other within a single  
query.


Some more below

On Feb 29, 2008, at 11:52 AM, [EMAIL PROTECTED] wrote:


Thanks for your response Grant.

You are right, depending of the language we could index the text in a
specific field. At request time, we would then ask all the fields  
for the

query.

I see however a few possible problems with this approach. By order of
decreasing importance:

- Influence on relevance

I assume the idf is calculated on a field by field basis? In the  
context of
one field per language, the documents whose language is the less  
present in

the index will receive an unusual boost for cross-lingual tokens. This
situation can be quite frequent as the distribution of languages in  
the
index is usually heterogeneous. Even if it was homogeneous, we would  
have

the problem with rare text in one language citing words in another.

On the other hand, you are right in the sense that the idf of language
specific words is also altered. In the context of one field for all
languages, the idf could be very low for a word if it is a common  
word in
another language. For example, the world "thé" in French is quite  
rare, but

its idf would be greatly altered by the word "the" in English.


I would be interesting to see a study on a real index about the  
effects of this.






We have a dilemma here...

- Performance

Queries are in O(log n) if I'm not mistaken? Then a disjunction  
query on x

language fields would be nearly x times slower, no?


Again, I think this depends on how you setup the indices, etc.




- Verbose configuration

Not an important point, but with the dynamic field type, you  
configure only
one time all the languages. Otherwise, you must do so for each text  
field.


The query handler configuration would also be much more verbose. We  
usually

use the dismax handler and the qf could become very long.


True.





- Highlight

Not an important point either, but a bit of work need to be done to
aggregate the results.

In conclusion, the choice is not so clear for me. Your remark on the
relevance made me think a bit more on multilingual problems. There  
may be a

way to tune the idf of some fields depending on others?

Another idea would be to boost documents in the language of the  
request.

This may be actually much simpler.

If you have any idea on the subject I'm very interested!

Nicolas


-Message d'origine-
De : Grant Ingersoll [mailto:[EMAIL PROTECTED]
Envoyé : vendredi 29 février 2008 14:06
À : solr-user@lucene.apache.org
Objet : Re: Proposition of a new feature: Dynamic Field Types

Why can't you choose the proper field in your application and keep
separate fields per language?  Putting them all in the same field,
regardless of language, is not a good idea in my opinion because it is
more than likely going to skew your statistics and lower your  
relevance.


That being said, the dynamic field type is still an interesting idea.

-Grant

On Feb 29, 2008, at 5:56 AM, [EMAIL PROTECTED] wrote:

Dynamic field types are field types that act as proxies to other  
field

types. The choice of the field type to use is done on a per document
basis
and is dependent of the values of the document's fields.

The use case that led us to this feature is the indexation of
documents in
different languages. We use a specific analyzer for each language
but want
to index semantic information that is not specific to the language.

For example, we would add in the index the semantic tag {co:Paris}
for the
expressions "Paris", "capital city of France", "the city of lights"  
in

English and "Paris", "capitale de la France", "la ville lumière" in
French.
This allows us to provide advanced functionalities such as semantic
and
cross-lingual search.

To do so in SOLR, we chose to index texts written in different
languages in
the same field, while analyzing them with different analyzers. Hence
the
proposition of a new feature that respond to this need: Dynamic
Field Types.

The idea of this new field type is to act as a proxy to other field
types.
Depending of the values of some fields of the document to index, it
chooses
the correct field type to use. In our situation, we use it to choose
the
correct language dependent field type 

Re: Proposition of a new feature: Dynamic Field Types

[Otis Gospodnetic]

I don't quite follow everything here (examples?), but I believe IDF of a term 
is not a per-field value, but "index-wide".  Does that change the arguments for 
this proposal then?

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

- Original Message 
> From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> To: solr-user@lucene.apache.org
> Sent: Friday, February 29, 2008 11:52:07 AM
> Subject: RE: Proposition of a new feature: Dynamic Field Types
> 
> Thanks for your response Grant.
> 
> You are right, depending of the language we could index the text in a
> specific field. At request time, we would then ask all the fields for the
> query.
> 
> I see however a few possible problems with this approach. By order of
> decreasing importance:
> 
> - Influence on relevance
> 
> I assume the idf is calculated on a field by field basis? In the context of
> one field per language, the documents whose language is the less present in
> the index will receive an unusual boost for cross-lingual tokens. This
> situation can be quite frequent as the distribution of languages in the
> index is usually heterogeneous. Even if it was homogeneous, we would have
> the problem with rare text in one language citing words in another.
> 
> On the other hand, you are right in the sense that the idf of language
> specific words is also altered. In the context of one field for all
> languages, the idf could be very low for a word if it is a common word in
> another language. For example, the world "thé" in French is quite rare, but
> its idf would be greatly altered by the word "the" in English.
> 
> We have a dilemma here...
> 
> - Performance
> 
> Queries are in O(log n) if I'm not mistaken? Then a disjunction query on x
> language fields would be nearly x times slower, no?
> 
> - Verbose configuration
> 
> Not an important point, but with the dynamic field type, you configure only
> one time all the languages. Otherwise, you must do so for each text field.
> 
> The query handler configuration would also be much more verbose. We usually
> use the dismax handler and the qf could become very long.
> 
> - Highlight
> 
> Not an important point either, but a bit of work need to be done to
> aggregate the results.
> 
> In conclusion, the choice is not so clear for me. Your remark on the
> relevance made me think a bit more on multilingual problems. There may be a
> way to tune the idf of some fields depending on others?
> 
> Another idea would be to boost documents in the language of the request.
> This may be actually much simpler.
> 
> If you have any idea on the subject I'm very interested!
> 
> Nicolas
> 
> 
> -Message d'origine-
> De : Grant Ingersoll [mailto:[EMAIL PROTECTED] 
> Envoyé : vendredi 29 février 2008 14:06
> À : solr-user@lucene.apache.org
> Objet : Re: Proposition of a new feature: Dynamic Field Types
> 
> Why can't you choose the proper field in your application and keep  
> separate fields per language?  Putting them all in the same field,  
> regardless of language, is not a good idea in my opinion because it is  
> more than likely going to skew your statistics and lower your relevance.
> 
> That being said, the dynamic field type is still an interesting idea.
> 
> -Grant
> 
> On Feb 29, 2008, at 5:56 AM, [EMAIL PROTECTED] wrote:
> 
> > Dynamic field types are field types that act as proxies to other field
> > types. The choice of the field type to use is done on a per document  
> > basis
> > and is dependent of the values of the document's fields.
> >
> > The use case that led us to this feature is the indexation of  
> > documents in
> > different languages. We use a specific analyzer for each language  
> > but want
> > to index semantic information that is not specific to the language.
> >
> > For example, we would add in the index the semantic tag {co:Paris}  
> > for the
> > expressions "Paris", "capital city of France", "the city of lights" in
> > English and "Paris", "capitale de la France", "la ville lumière" in  
> > French.
> > This allows us to provide advanced functionalities such as semantic  
> > and
> > cross-lingual search.
> >
> > To do so in SOLR, we chose to index texts written in different  
> > languages in
> > the same field, while analyzing them with different analyzers. Hence  
> > the
> > proposition of a new feature that respond to this need: Dynamic  
> > Field Types.
> >
> > The idea of this new field type is to act 

Re: Proposition of a new feature: Dynamic Field Types

[Yonik Seeley]

On Sat, Mar 1, 2008 at 9:38 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> I don't quite follow everything here (examples?), but I believe IDF of a term 
> is not a per-field value, but "index-wide".

I think Nicolas meant that idfs are field specific, and that is the
case (index-wide, per field).

-Yonik

RE: Proposition of a new feature: Dynamic Field Types

[nicolas . dessaigne]

> How many languages are you dealing with?

The number of languages depends greatly on the project. We are usually
dealing with 2 or 3 languages and I've yet to see a project with more than
5.

> How are you generating your queries?

With a specific handler (based on the DisMax) that have an extra lng
parameter to know which analyzer to use.

> Are you taking your source language and translating it to each of the
languages?  Or are you just targeting one source to destination language?

No translation involved. We actually infer semantic information independent
of the language and use it in our cross-lingual queries: one source language
for the query and many destination languages for the results.

As discussed earlier, you're right about inadequacy of cross-lingual search
for correct ranking. It seems there is no perfect solution for this need,
just tradeoffs...

We'll do a test with the two approaches (one field for all / one field per
language) and I'll report back when we've made a choice.

> Again, I think this depends on how you setup the indices, etc.

About the performance question, I meant that for fields configured in the
same manner (except for the language supported by the analyzer of course)
and for a big number of documents, the time consumed by a query on one field
containing a value for all the documents would be nearly half of the
cumulative time consumed by two queries each on a partition of the
documents. This is due to the O(log n) complexity of queries: a query on a
index of 10,000,000 docs is only slightly slower than the same query on a
index of 5,000,000 docs.

Thanks for your advice,
I'll try to report back on our tests.

Nicolas

-Message d'origine-
De : Grant Ingersoll [mailto:[EMAIL PROTECTED] 
Envoyé : samedi 1 mars 2008 13:54
À : solr-user@lucene.apache.org
Objet : Re: Proposition of a new feature: Dynamic Field Types

How many languages are you dealing with?  How are you generating your  
queries?  Are you taking your source language and translating it to  
each of the languages?  Or are you just targeting one source to  
destination language?  Back when I was doing CLIR, I would create  
separate indexes, but that is not to say it is appropriate for your  
task.  It was always highly suspect in my mind the notion of combining  
the results of multiple languages into a single hit list.  I just  
doubt that the scores are meaningful in that way since scores should  
generally only be considered relative to each other within a single  
query.

Some more below

On Feb 29, 2008, at 11:52 AM, [EMAIL PROTECTED] wrote:

> Thanks for your response Grant.
>
> You are right, depending of the language we could index the text in a
> specific field. At request time, we would then ask all the fields  
> for the
> query.
>
> I see however a few possible problems with this approach. By order of
> decreasing importance:
>
> - Influence on relevance
>
> I assume the idf is calculated on a field by field basis? In the  
> context of
> one field per language, the documents whose language is the less  
> present in
> the index will receive an unusual boost for cross-lingual tokens. This
> situation can be quite frequent as the distribution of languages in  
> the
> index is usually heterogeneous. Even if it was homogeneous, we would  
> have
> the problem with rare text in one language citing words in another.
>
> On the other hand, you are right in the sense that the idf of language
> specific words is also altered. In the context of one field for all
> languages, the idf could be very low for a word if it is a common  
> word in
> another language. For example, the world "thé" in French is quite  
> rare, but
> its idf would be greatly altered by the word "the" in English.

I would be interesting to see a study on a real index about the  
effects of this.


>
>
> We have a dilemma here...
>
> - Performance
>
> Queries are in O(log n) if I'm not mistaken? Then a disjunction  
> query on x
> language fields would be nearly x times slower, no?

Again, I think this depends on how you setup the indices, etc.

>
>
> - Verbose configuration
>
> Not an important point, but with the dynamic field type, you  
> configure only
> one time all the languages. Otherwise, you must do so for each text  
> field.
>
> The query handler configuration would also be much more verbose. We  
> usually
> use the dismax handler and the qf could become very long.

True.


>
>
> - Highlight
>
> Not an important point either, but a bit of work need to be done to
> aggregate the results.
>
> In conclusion, the choice is not so clear for me. Your remark on the
> relevance made me think a bit more on multilingual problems. There  
> may be a
> way to tune the i

RE: Proposition of a new feature: Dynamic Field Types

[nicolas . dessaigne]

You're right Yonik, that's what I meant.

As for an example, let us place ourselves in two situations:
- Situation A: one field ("text") where the texts of all documents are
indexed;
- Situation B: one field per language ("text_en", "text_fr", etc.) where
only the texts of documents in that language are indexed.

Now, take a query with the French word "voiture" (means car). There are of
course much more French documents than English ones containing this word. In
situation A, the scores of documents will only depends on the tf of
"voiture". French and English documents will be treated equally. In
situation B, the idf of "voiture" in the field "text_en" will be very
important due to the rarity of this word in English. English documents will
then always be ranked higher.

Then, take a query with the English word "car". This word is very common in
French (means "because" but can also be used for other meanings). In
situation A, French documents will be ranked higher because of their high
tf. Situation B is better in this case as the frequency of "car" in French
is not taken in account in the ranking. The score of English documents will
be calculated normally while French documents will be ranked lower due to
the very low idf (or even not appear if stop words are used). 

So far, situation B with a boost for documents in the same language as the
query seems the more promising one (at least when dealing with the relevance
problem).

Another interesting solution is to use language dependant stop words in
situation A to avoid altering idf for other language. But this incurs other
problems (stop words are not always desirable).

I hope this example clarifies the identified problem of cross-lingual
retrieval in Lucene. Sorry if it is not as clear as I would like (English is
not my mother tongue).

Nicolas

-Message d'origine-
De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de Yonik Seeley
Envoyé : dimanche 2 mars 2008 03:45
À : solr-user@lucene.apache.org
Objet : Re: Proposition of a new feature: Dynamic Field Types

On Sat, Mar 1, 2008 at 9:38 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> I don't quite follow everything here (examples?), but I believe IDF of a
term is not a per-field value, but "index-wide".

I think Nicolas meant that idfs are field specific, and that is the
case (index-wide, per field).

-Yonik

Unable to filter fq param on a dynamic field

[Barry Harding]

Hi,

I have a fairly simple solr setup with several predefined fields that are 
indexed and stored and also depending on the type of product I also add various 
dynamic fields of type string to a record, and I should mention that I am using 
the
solr.DisMaxRequestHandler request handler called "/IvolutionSearch" in my 
example requests.



My Schema is as follows:







































Now I can query for any of the fixed field types Such as ProductName or 
ReviewRating and get the results I expect but when I try to run a filter query 
on the dynamic fields in the result, I always end up with no results being 
returned.



So if I run the following query against my copy of solr 1.3 I get the results I 
am expecting



http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100



- 
- 
  $A
  Mono Laser Printers
  Printers|Mono Laser Printers
  Wired
  UK
  UK$AQ969719
  3500V_DN
  Xerox
  Xerox
  Monochrome
  The Xerox Phaser 3500 series printer provides an 
affordable solution to meet the increasing volume a
  464.10
  Q969719
  XEROX 3500DN MONO LASER
  
  E000
  2
  Laser
  26099.jpg
  Workgroup printer
  MLASERPRN
  2008-09-17T17:10:44.37Z
  
- 
  $B
  Mono Laser Printers
  Printers|Mono Laser Printers
  Wired

and so on for the 100 results

no if I try to filter those results to just those that contain 
"Output-Type-facet" equaling "Monochrome"

using :
 
http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq=Output-Type-facet:Monochrome
or

http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq="Output-Type-facet":"Monochrome";
or
http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq="Output-Type-facet:Monochrome";
or
http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq=Output-Type-facet:"Monochrome";



I just get zero results back even though I know that filed contains that value, 
please before I pull my hair out tell me what mistake I have made, why can I 
query using a static field and not a dynamic field

any help even if its to say I have been stupid or to tell me to reread a 
section of the manual/Wiki because I did not get the point much appreciated.



Thanks

Barry H<mailto:[EMAIL PROTECTED]>










Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number 
114143.  Registered Office: Caledonian Exchange, 19a Canning Street, Edinburgh 
EH3 8EG.  Telephone +44 (0)1933 686000.

Re: [2] Saving dynamic field name without dynamic extension

[Debra]

Thank you, your suggestion looks like the way to go...


Mike Klaas wrote:
> 
> On 3/6/07, Debra <[EMAIL PROTECTED]> wrote:
>>
>> I want to add a suffix to my fields names to use the dynamic fields
>> feature.
>> Is there a way to save the field name without the suffix so users can
>> search
>> by field  with plain field name?
> 
> No, and I'm not sure that it is possible.  Solr needs to know the type
> of a field at all times--not just during indexing.
> 
> Why not create a _user suffix, and programmatically add the suffix to
> user queries before it reaches solr?
> 
> -Mike
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Saving-dynamic-field-name-without-dynamic-extension-tf3358269.html#a9343182
Sent from the Solr - User mailing list archive at Nabble.com.

Problem in parsing non-string dynamic field by using IndexReader

[Yuchen Wang]

I have a task to parse all documents in a solr index. I use Lucene
IndexReader to read the index and go through each field from all documents.
However, for float or int dynamic fields, the stringValue() call always
returns some special characters. I tried tokenStreamValue, byteValue,
readerValue, and they return null.
Following is my method to parse the solr index. My question is, how can I
get the values from non-string dynamic fields properly?

public static void main(String[] args) throws Exception {
IndexReader reader =
IndexReader.open("/path/to/my/index/directory");

int total = reader.numDocs();
System.out.println("Total documents: " + total);

for (int i = 0; i < 1; i++) {
Document d = reader.document(i);

List fields = d.getFields();

for (Field f : fields) {
String name = f.name();
String val = f.stringValue();

   System.out.println("get field / value: [" + name + "=" + val
+ "]");}
}

reader.close();
}

Re: How to Create dynamic field names using script transformers

[Erik Hatcher]

Barani -

Give us some details of what you tried, what you expected to happen,  
and what actually happened.


Erik


On Jan 26, 2010, at 4:15 PM, JavaGuy84 wrote:



Hi,

I am trying to generate a dynamic fieldname using custom  
transformers but

couldn't achieve the expected results.

My requirement is that I do not want to hardcode some of field names  
used by
SOLR for indexing, instead the field name should be generated using  
the data

retreieved from a table.

Any help on this regard is greatly appreciated.

Thanks,
Barani
--
View this message in context: 
http://old.nabble.com/How-to-Create-dynamic-field-names-using-script-transformers-tp27329876p27329876.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Create dynamic field names using script transformers

[JavaGuy84]

Hey Erik,

Thanks a lot for your reply.. I am a newbie to SOLR ...

I am just trying to use the example present in Apache WIKI to understand
"how" the scriptTransformer works. I want to know how to pass the data from
table.field to transformer and get back the data from transformer and set
the value to any field.


 <![CDATA[
 function f1(row){
 row.put('Hello', 'Test');
 return row;
 }
 ]]>
 

 
 
 
 



Basically I want a field like... 
and index this field so that users can search on this dynamic field and get
the corresponding data also.


Thanks,
Barani



Erik Hatcher-4 wrote:
> 
> Barani -
> 
> Give us some details of what you tried, what you expected to happen,  
> and what actually happened.
> 
>   Erik
> 
> 
> On Jan 26, 2010, at 4:15 PM, JavaGuy84 wrote:
> 
>>
>> Hi,
>>
>> I am trying to generate a dynamic fieldname using custom  
>> transformers but
>> couldn't achieve the expected results.
>>
>> My requirement is that I do not want to hardcode some of field names  
>> used by
>> SOLR for indexing, instead the field name should be generated using  
>> the data
>> retreieved from a table.
>>
>> Any help on this regard is greatly appreciated.
>>
>> Thanks,
>> Barani
>> -- 
>> View this message in context:
>> http://old.nabble.com/How-to-Create-dynamic-field-names-using-script-transformers-tp27329876p27329876.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: 
http://old.nabble.com/How-to-Create-dynamic-field-names-using-script-transformers-tp27329876p27330330.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to Create dynamic field names using script transformers

[JavaGuy84]

To add some more details, this is what I am trying to acheive...

There are 2 fields present in a database table and I am trying to make those
2 fields as key value pair.

Eg: Consider there are 2 fields associated with each other (Propertyid and
propertyValue)

I want the property id as field name and property value as its field
value..something like...

<111>Test<1>

Thanks,
Barani


JavaGuy84 wrote:
> 
> Hi,
> 
> I am trying to generate a dynamic fieldname using custom transformers but
> couldn't achieve the expected results.
> 
> My requirement is that I do not want to hardcode some of field names used
> by SOLR for indexing, instead the field name should be generated using the
> data retreieved from a table.
> 
> Any help on this regard is greatly appreciated.
> 
> Thanks,
> Barani
> 

-- 
View this message in context: 
http://old.nabble.com/How-to-Create-dynamic-field-names-using-script-transformers-tp27329876p27330470.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Unable to filter fq param on a dynamic field

[Otis Gospodnetic]

Barry, does this return the correct hits:

http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=Output-Type-facet:Monochrome

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Barry Harding <[EMAIL PROTECTED]>
> To: "solr-user@lucene.apache.org" 
> Sent: Thursday, September 18, 2008 7:21:49 AM
> Subject: Unable to filter fq param on a dynamic field
> 
> 
> 
> Hi,
> 
> I have a fairly simple solr setup with several predefined fields that are 
> indexed and stored and also depending on the type of product I also add 
> various 
> dynamic fields of type string to a record, and I should mention that I am 
> using 
> the
> solr.DisMaxRequestHandler request handler called "/IvolutionSearch" in my 
> example requests.
> 
> 
> 
> My Schema is as follows:
> 
> 
> required="true" />
> 
> 
> required="true" />
> 
> 
> required="true" />
> 
> 
> required="true" />
> 
> 
> 
> 
> required="true" />
> 
> 
> />
> 
> 
> required="true" />
> 
> 
> />
> 
> 
> 
> 
> required="true" />
> 
> 
> required="false" />
> 
> 
> required="false" />
> 
> 
> required="true" />
> 
> 
> required="false" />
> 
> 
> required="true" />
> 
> 
> 
> 
> multiValued="false" />
> 
> 
> 
> Now I can query for any of the fixed field types Such as ProductName or 
> ReviewRating and get the results I expect but when I try to run a filter 
> query 
> on the dynamic fields in the result, I always end up with no results being 
> returned.
> 
> 
> 
> So if I run the following query against my copy of solr 1.3 I get the results 
> I 
> am expecting
> 
> 
> 
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100
> 
> 
> 
> - 
> - 
>   $A
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
>   UK
>   UK$AQ969719
>   3500V_DN
>   Xerox
>   Xerox
>   Monochrome
>   The Xerox Phaser 3500 series printer provides an 
> affordable solution to meet the increasing volume a
>   464.10
>   Q969719
>   XEROX 3500DN MONO LASER
>   
>   E000
>   2
>   Laser
>   26099.jpg
>   Workgroup printer
>   MLASERPRN
>   2008-09-17T17:10:44.37Z
>   
> - 
>   $B
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
> 
> and so on for the 100 results
> 
> no if I try to filter those results to just those that contain 
> "Output-Type-facet" equaling "Monochrome"
> 
> using :
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq=Output-Type-facet:Monochrome
> or
> 
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq="Output-Type-facet":"Monochrome";
> or
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq="Output-Type-facet:Monochrome";
> or
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq=Output-Type-facet:"Monochrome";
> 
> 
> 
> I just get zero results back even though I know that filed contains that 
> value, 
> please before I pull my hair out tell me what mistake I have made, why can I 
> query using a static field and not a dynamic field
> 
> any help even if its to say I have been stupid or to tell me to reread a 
> section 
> of the manual/Wiki because I did not get the point much appreciated.
> 
> 
> 
> Thanks
> 
> Barry H
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number 
> 114143.  Registered Office: Caledonian Exchange, 19a Canning Street, 
> Edinburgh 
> EH3 8EG.  Telephone +44 (0)1933 686000.

RE: Unable to filter fq param on a dynamic field

[Barry Harding]

Hi Otis,

no that does not seem to bring back the correct results either in fact its 
still zero results.

Its also not bringing back results if I use the standard handler
 http://127.0.0.1:8080/apache-solr-1.3.0/select?q=Output-Type-facet:Monochrome

but the field is visible in the documents returned if I search for the 
following:

http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser

so i know that the field is in the results generated (shown below)

 
- 
- 
  0
  666
- 
  laser
  
  
- 
- 
  $A
  Mono Laser Printers
  Printers|Mono Laser Printers
  Wired
  UK
  UK$AQ63360
  Q7697A#ABU
  Hewlett Packard
  HP
  Monochrome
  The LaserJet 9000 series printer is HP's fastest, most 
versatile LaserJet designed for today's distr
  1388.99
  Q63360
  HP LASERJET 9040 MONO LASER
  
  E000
  2
  Laser
  98404.jpg
  Workgroup printer
  MLASERPRN
  2008-09-18T16:44:01.029Z
  
- 
  $B
  Mono Laser Printers
  Printers|Mono Laser Printers
  Wired




Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number 
114143.  Registered Office: Caledonian Exchange, 19a Canning Street, Edinburgh 
EH3 8EG.  Telephone +44 (0)1933 686000.

Re: Unable to filter fq param on a dynamic field

[Otis Gospodnetic]

Barry,

You are seeing the value of the field as it was saved (as the original), but 
perhaps something is funky with how it was analyzed/tokenized at search time 
and how it is being analyzed now at query time.  Double-check your 
fieldType/analysis settings for this field and make sure you are using the 
same/compatible analyzers at both index and query time.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Barry Harding <[EMAIL PROTECTED]>
> To: "solr-user@lucene.apache.org" 
> Sent: Thursday, September 18, 2008 12:53:08 PM
> Subject: RE: Unable to filter fq param on a dynamic field
> 
> Hi Otis,
> 
> no that does not seem to bring back the correct results either in fact its 
> still 
> zero results.
> 
> Its also not bringing back results if I use the standard handler
> http://127.0.0.1:8080/apache-solr-1.3.0/select?q=Output-Type-facet:Monochrome
> 
> but the field is visible in the documents returned if I search for the 
> following:
> 
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser
> 
> so i know that the field is in the results generated (shown below)
> 
> 
> -
> -
> name="responseHeader">
>   0
>   666
> -
> name="params">
>   laser
>   
>   
> -
> name="response" numFound="8056" start="0">
> -
>   $A
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
>   UK
>   UK$AQ63360
>   Q7697A#ABU
>   Hewlett Packard
>   HP
>   Monochrome
>   The LaserJet 9000 series printer is HP's fastest, most 
> versatile LaserJet designed for today's distr
>   1388.99
>   Q63360
>   HP LASERJET 9040 MONO LASER
>   
>   E000
>   2
>   Laser
>   98404.jpg
>   Workgroup printer
>   MLASERPRN
>   2008-09-18T16:44:01.029Z
>   
> -
>   $B
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
> 
> 
> 
> 
> Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number 
> 114143.  Registered Office: Caledonian Exchange, 19a Canning Street, 
> Edinburgh 
> EH3 8EG.  Telephone +44 (0)1933 686000.

RE: Unable to filter fq param on a dynamic field

[Barry Harding]

You were correct, I assumed that as I did not want to break the values in the 
fields into seperate tokens that I could just use "string" as the type in the 
schema for these fields.

As soon as I switched to a custom field type using

solr.KeywordTokenizerFactory and reindexed it all started working.





Thanks again for your help



Barry


From: Otis Gospodnetic [EMAIL PROTECTED]
Sent: 18 September 2008 20:40
To: solr-user@lucene.apache.org
Subject: Re: Unable to filter fq param on a dynamic field

Barry,

You are seeing the value of the field as it was saved (as the original), but 
perhaps something is funky with how it was analyzed/tokenized at search time 
and how it is being analyzed now at query time.  Double-check your 
fieldType/analysis settings for this field and make sure you are using the 
same/compatible analyzers at both index and query time.


Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Barry Harding <[EMAIL PROTECTED]>
> To: "solr-user@lucene.apache.org" 
> Sent: Thursday, September 18, 2008 12:53:08 PM
> Subject: RE: Unable to filter fq param on a dynamic field
>
> Hi Otis,
>
> no that does not seem to bring back the correct results either in fact its 
> still
> zero results.
>
> Its also not bringing back results if I use the standard handler
> http://127.0.0.1:8080/apache-solr-1.3.0/select?q=Output-Type-facet:Monochrome
>
> but the field is visible in the documents returned if I search for the
> following:
>
> http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser
>
> so i know that the field is in the results generated (shown below)
>
>
> -
> -
> name="responseHeader">
>   0
>   666
> -
> name="params">
>   laser
>
>
> -
> name="response" numFound="8056" start="0">
> -
>   $A
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
>   UK
>   UK$AQ63360
>   Q7697A#ABU
>   Hewlett Packard
>   HP
>   Monochrome
>   The LaserJet 9000 series printer is HP's fastest, most
> versatile LaserJet designed for today's distr
>   1388.99
>   Q63360
>   HP LASERJET 9040 MONO LASER
>
>   E000
>   2
>   Laser
>   98404.jpg
>   Workgroup printer
>   MLASERPRN
>   2008-09-18T16:44:01.029Z
>
> -
>   $B
>   Mono Laser Printers
>   Printers|Mono Laser Printers
>   Wired
>
>
>
> 
> Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number
> 114143.  Registered Office: Caledonian Exchange, 19a Canning Street, Edinburgh
> EH3 8EG.  Telephone +44 (0)1933 686000.



Misco is a division of Systemax Europe Ltd.  Registered in Scotland Number 
114143.  Registered Office: Caledonian Exchange, 19a Canning Street, Edinburgh 
EH3 8EG.  Telephone +44 (0)1933 686000.

Re: Unable to filter fq param on a dynamic field

[Chris Hostetter]

: Now I can query for any of the fixed field types Such as ProductName or 
: ReviewRating and get the results I expect but when I try to run a filter 
: query on the dynamic fields in the result, I always end up with no 
: results being returned.

from what i can tell based on this info you provided, this certianly seems 
like it should work...

:  
http://127.0.0.1:8080/apache-solr-1.3.0/IvolutionSearch?q=laser&rows=100&fq=Output-Type-facet:Monochrome

...what happens if you try this...

http://127.0.0.1:8080/apache-solr-1.3.0/select/?q=Output-Type-facet:Monochrome&debugQuery=true

PS...

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to 
an existing message, instead start a fresh email.  Even if you change the 
subject line of your email, other mail headers still track which thread 
you replied to and your question is "hidden" in that thread and gets less 
attention.   It makes following discussions in the mailing list archives 
particularly difficult.
See Also:  http://en.wikipedia.org/wiki/Thread_hijacking



-Hoss

Re: Problem in parsing non-string dynamic field by using IndexReader

[Otis Gospodnetic]

Yuchen,

Make sure the fields you are trying to read are stored (stored="true" in 
schema.xml)

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Yuchen Wang 
> To: solr-user@lucene.apache.org
> Sent: Sunday, July 5, 2009 12:43:50 AM
> Subject: Problem in parsing non-string dynamic field by using IndexReader
> 
> I have a task to parse all documents in a solr index. I use Lucene
> IndexReader to read the index and go through each field from all documents.
> However, for float or int dynamic fields, the stringValue() call always
> returns some special characters. I tried tokenStreamValue, byteValue,
> readerValue, and they return null.
> Following is my method to parse the solr index. My question is, how can I
> get the values from non-string dynamic fields properly?
> 
> public static void main(String[] args) throws Exception {
> IndexReader reader =
> IndexReader.open("/path/to/my/index/directory");
> 
> int total = reader.numDocs();
> System.out.println("Total documents: " + total);
> 
> for (int i = 0; i < 1; i++) {
> Document d = reader.document(i);
> 
> Listfields = d.getFields();
> 
> for (Field f : fields) {
> String name = f.name();
> String val = f.stringValue();
> 
>System.out.println("get field / value: [" + name + "=" + val
> + "]");}
> }
> 
> reader.close();
> }

Re: Problem in parsing non-string dynamic field by using IndexReader

[Yuchen Wang]

Thanks for the reply. However, in the code I posted, where should I load the
schema.xml? I just created a Lucene IndexReader directly.

On Sun, Jul 5, 2009 at 9:31 AM, Otis Gospodnetic  wrote:

>
> Yuchen,
>
> Make sure the fields you are trying to read are stored (stored="true" in
> schema.xml)
>
>  Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message 
> > From: Yuchen Wang 
> > To: solr-user@lucene.apache.org
> > Sent: Sunday, July 5, 2009 12:43:50 AM
> > Subject: Problem in parsing non-string dynamic field by using IndexReader
> >
> > I have a task to parse all documents in a solr index. I use Lucene
> > IndexReader to read the index and go through each field from all
> documents.
> > However, for float or int dynamic fields, the stringValue() call always
> > returns some special characters. I tried tokenStreamValue, byteValue,
> > readerValue, and they return null.
> > Following is my method to parse the solr index. My question is, how can I
> > get the values from non-string dynamic fields properly?
> >
> > public static void main(String[] args) throws Exception {
> > IndexReader reader =
> > IndexReader.open("/path/to/my/index/directory");
> >
> > int total = reader.numDocs();
> > System.out.println("Total documents: " + total);
> >
> > for (int i = 0; i < 1; i++) {
> > Document d = reader.document(i);
> >
> > Listfields = d.getFields();
> >
> > for (Field f : fields) {
> > String name = f.name();
> > String val = f.stringValue();
> >
> >System.out.println("get field / value: [" + name + "=" +
> val
> > + "]");}
> > }
> >
> > reader.close();
> > }
>
>

Re: Problem in parsing non-string dynamic field by using IndexReader

[Otis Gospodnetic]

Yuchen,

schema.xml is a Solr configuration file that you can find in a conf directory 
under Solr home.  Please go through the Solr tutorial on the site first.

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
> From: Yuchen Wang 
> To: solr-user@lucene.apache.org
> Sent: Sunday, July 5, 2009 1:19:12 PM
> Subject: Re: Problem in parsing non-string dynamic field by using IndexReader
> 
> Thanks for the reply. However, in the code I posted, where should I load the
> schema.xml? I just created a Lucene IndexReader directly.
> 
> On Sun, Jul 5, 2009 at 9:31 AM, Otis Gospodnetic 
> > wrote:
> 
> >
> > Yuchen,
> >
> > Make sure the fields you are trying to read are stored (stored="true" in
> > schema.xml)
> >
> >  Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > - Original Message 
> > > From: Yuchen Wang 
> > > To: solr-user@lucene.apache.org
> > > Sent: Sunday, July 5, 2009 12:43:50 AM
> > > Subject: Problem in parsing non-string dynamic field by using IndexReader
> > >
> > > I have a task to parse all documents in a solr index. I use Lucene
> > > IndexReader to read the index and go through each field from all
> > documents.
> > > However, for float or int dynamic fields, the stringValue() call always
> > > returns some special characters. I tried tokenStreamValue, byteValue,
> > > readerValue, and they return null.
> > > Following is my method to parse the solr index. My question is, how can I
> > > get the values from non-string dynamic fields properly?
> > >
> > > public static void main(String[] args) throws Exception {
> > > IndexReader reader =
> > > IndexReader.open("/path/to/my/index/directory");
> > >
> > > int total = reader.numDocs();
> > > System.out.println("Total documents: " + total);
> > >
> > > for (int i = 0; i < 1; i++) {
> > > Document d = reader.document(i);
> > >
> > > Listfields = d.getFields();
> > >
> > > for (Field f : fields) {
> > > String name = f.name();
> > > String val = f.stringValue();
> > >
> > >System.out.println("get field / value: [" + name + "=" +
> > val
> > > + "]");}
> > > }
> > >
> > > reader.close();
> > > }
> >
> >

Re: Problem in parsing non-string dynamic field by using IndexReader

[Yuchen Wang]

OK, here is my latest code to get the IndexReader from the solr core.
However, it still printed out the non-string fields as special chars. I do
use the schema file here. Please help.
public static void main(String[] args) throws Exception {
SolrConfig config = new SolrConfig("/Users/yuchen/Work/data/",
"solrconfig.xml", null);
IndexSchema schema = new IndexSchema(config, "schema.xml", null);

CoreContainer container = new CoreContainer(new
SolrResourceLoader(SolrResourceLoader.locateInstanceDir()));
CoreDescriptor dcore = new CoreDescriptor(container, "solr0",
config.getResourceLoader().getInstanceDir());
dcore.setConfigName(config.getResourceName());
dcore.setSchemaName(schema.getResourceName());
SolrCore core = new SolrCore("solr0", "/Users/yuchen/Work/data",
config, schema, dcore);
container.register("solr0", core, false);

IndexReader reader = core.getSearcher().get().getReader();

FieldCache cache = FieldCache.DEFAULT;

int total = reader.numDocs();
System.out.println("Total documents: " + total);

for (int i = 0; i < 1; i++) {
System.out.println("\n=== Got Node: " + i + "
=");
Document d = reader.document(i);

List fields = d.getFields();

for (Field f : fields) {
String name = f.name();
String val = f.stringValue();
System.out.println("get field / value: [" + name + "=" + val
+ "]");
}
}

reader.close();
}


On Sun, Jul 5, 2009 at 7:58 PM, Otis Gospodnetic  wrote:

>
> Yuchen,
>
> schema.xml is a Solr configuration file that you can find in a conf
> directory under Solr home.  Please go through the Solr tutorial on the site
> first.
>
>  Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> - Original Message 
> > From: Yuchen Wang 
> > To: solr-user@lucene.apache.org
> > Sent: Sunday, July 5, 2009 1:19:12 PM
> > Subject: Re: Problem in parsing non-string dynamic field by using
> IndexReader
> >
> > Thanks for the reply. However, in the code I posted, where should I load
> the
> > schema.xml? I just created a Lucene IndexReader directly.
> >
> > On Sun, Jul 5, 2009 at 9:31 AM, Otis Gospodnetic
> > > wrote:
> >
> > >
> > > Yuchen,
> > >
> > > Make sure the fields you are trying to read are stored (stored="true"
> in
> > > schema.xml)
> > >
> > >  Otis
> > > --
> > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > >
> > >
> > >
> > > - Original Message 
> > > > From: Yuchen Wang
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Sunday, July 5, 2009 12:43:50 AM
> > > > Subject: Problem in parsing non-string dynamic field by using
> IndexReader
> > > >
> > > > I have a task to parse all documents in a solr index. I use Lucene
> > > > IndexReader to read the index and go through each field from all
> > > documents.
> > > > However, for float or int dynamic fields, the stringValue() call
> always
> > > > returns some special characters. I tried tokenStreamValue, byteValue,
> > > > readerValue, and they return null.
> > > > Following is my method to parse the solr index. My question is, how
> can I
> > > > get the values from non-string dynamic fields properly?
> > > >
> > > > public static void main(String[] args) throws Exception {
> > > > IndexReader reader =
> > > > IndexReader.open("/path/to/my/index/directory");
> > > >
> > > > int total = reader.numDocs();
> > > > System.out.println("Total documents: " + total);
> > > >
> > > > for (int i = 0; i < 1; i++) {
> > > > Document d = reader.document(i);
> > > >
> > > > Listfields = d.getFields();
> > > >
> > > > for (Field f : fields) {
> > > > String name = f.name();
> > > > String val = f.stringValue();
> > > >
> > > >System.out.println("get field / value: [" + name + "="
> +
> > > val
> > > > + "]");}
> > > > }
> > > >
> > > > reader.close();
> > > > }
> > >
> > >
>
>

Re: Problem in parsing non-string dynamic field by using IndexReader

[Chris Hostetter]

: OK, here is my latest code to get the IndexReader from the solr core.
: However, it still printed out the non-string fields as special chars. I do
: use the schema file here. Please help.

you'll want to use the IndexSchema object to get the FieldType 
object for your field name.  then use the FieldType to convert the values 
in the index to readable values.

Take a look at the javadocs for IndexSearcher and FieldType for more 
details.  

if you look at code like the XMLResponseWriter you'll see examples of 
iterating over all the fields in a Document and using those methods.



-Hoss

Re: Problem in parsing non-string dynamic field by using IndexReader

[Yuchen Wang]

that works perfectly! Thanks a lot!

On Mon, Jul 6, 2009 at 2:12 PM, Chris Hostetter wrote:

> : OK, here is my latest code to get the IndexReader from the solr core.
> : However, it still printed out the non-string fields as special chars. I
> do
> : use the schema file here. Please help.
>
> you'll want to use the IndexSchema object to get the FieldType
> object for your field name.  then use the FieldType to convert the values
> in the index to readable values.
>
> Take a look at the javadocs for IndexSearcher and FieldType for more
> details.
>
> if you look at code like the XMLResponseWriter you'll see examples of
> iterating over all the fields in a Document and using those methods.
>
>
>
> -Hoss
>
>

how to scan dynamic field without specifying each field in query

[gdeconto]

say I have a dynamic field called Foo* (where * can be in the hundreds) and
want to search Foo* for a value of 3 (for example)

I know I can do this via this:

http://localhost:8994/solr/select?q=(Foo1:3 OR Foo2:3 OR Foo3:3 OR …
Foo999:3)

However, is there a better way?  i.e. is there some way to query by a
function I create, possibly something like this:

http://localhost:8994/solr/select?q=myfunction(‘Foo’, 3)

where myfunction itself iterates thru all the instances of Foo*

any help appreciated

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25280228.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: how to scan dynamic field without specifying each field in query

[Manepalli, Kalyan]

You can copy the dynamic fields value into a different field and query on that 
field.

Thanks,
Kalyan Manepalli

-Original Message-
From: gdeconto [mailto:gerald.deco...@topproducer.com] 
Sent: Thursday, September 03, 2009 12:06 PM
To: solr-user@lucene.apache.org
Subject: how to scan dynamic field without specifying each field in query


say I have a dynamic field called Foo* (where * can be in the hundreds) and
want to search Foo* for a value of 3 (for example)

I know I can do this via this:

http://localhost:8994/solr/select?q=(Foo1:3 OR Foo2:3 OR Foo3:3 OR ...
Foo999:3)

However, is there a better way?  i.e. is there some way to query by a
function I create, possibly something like this:

http://localhost:8994/solr/select?q=myfunction('Foo', 3)

where myfunction itself iterates thru all the instances of Foo*

any help appreciated

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25280228.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to scan dynamic field without specifying each field in query

[Avlesh Singh]

>
> I know I can do this via this: http://localhost:8994/solr/select?q=(Foo1:3OR 
> Foo2:3 OR Foo3:3 OR ... Foo999:3)
>
Careful! You may hit the upper limit for MAX_BOOLEAN_CLAUSES this way.


> You can copy the dynamic fields value into a different field and query on
> that field.
>
Good idea!

Cheers
**Avlesh

On Thu, Sep 3, 2009 at 10:47 PM, Manepalli, Kalyan <
kalyan.manepa...@orbitz.com> wrote:

> You can copy the dynamic fields value into a different field and query on
> that field.
>
> Thanks,
> Kalyan Manepalli
>
> -Original Message-
> From: gdeconto [mailto:gerald.deco...@topproducer.com]
> Sent: Thursday, September 03, 2009 12:06 PM
> To: solr-user@lucene.apache.org
> Subject: how to scan dynamic field without specifying each field in query
>
>
> say I have a dynamic field called Foo* (where * can be in the hundreds) and
> want to search Foo* for a value of 3 (for example)
>
> I know I can do this via this:
>
> http://localhost:8994/solr/select?q=(Foo1:3<http://localhost:8994/solr/select?q=%28Foo1:3>OR
>  Foo2:3 OR Foo3:3 OR ...
> Foo999:3)
>
> However, is there a better way?  i.e. is there some way to query by a
> function I create, possibly something like this:
>
> http://localhost:8994/solr/select?q=myfunction('Foo<http://localhost:8994/solr/select?q=myfunction%28%27Foo>',
> 3)
>
> where myfunction itself iterates thru all the instances of Foo*
>
> any help appreciated
>
> --
> View this message in context:
> http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25280228.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

RE: how to scan dynamic field without specifying each field in query

[gdeconto]

thx for the reply.

you mean into a multivalue field?  possible, but was wondering if there was
something more flexible than that.  the ability to use a function (ie
myfunction) would open up some possibilities for more complex searching and
search syntax.

I could write my own query parser with special extended syntax, but that is
farther than I wanted to go.



Manepalli, Kalyan wrote:
> 
> You can copy the dynamic fields value into a different field and query on
> that field.
> 
> Thanks,
> Kalyan Manepalli
> 
> 

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25280669.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to scan dynamic field without specifying each field in query

[Avlesh Singh]

A query parser, may be.
But that would not help either. End of the day, someone has to create those
many boolean queries in your case.

Cheers
Avlesh

On Thu, Sep 3, 2009 at 10:59 PM, gdeconto wrote:

>
> thx for the reply.
>
> you mean into a multivalue field?  possible, but was wondering if there was
> something more flexible than that.  the ability to use a function (ie
> myfunction) would open up some possibilities for more complex searching and
> search syntax.
>
> I could write my own query parser with special extended syntax, but that is
> farther than I wanted to go.
>
>
>
> Manepalli, Kalyan wrote:
> >
> > You can copy the dynamic fields value into a different field and query on
> > that field.
> >
> > Thanks,
> > Kalyan Manepalli
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25280669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: how to scan dynamic field without specifying each field in query

[Renaud Delbru]

Hi,

maybe SIREn [1] can help you for this task. SIREn is a Lucene plugin 
that allows to index and query tabular data. You can for example create 
a SIREn field "foo", index n values in n cells, and then query a 
specific cell or a range of cells. Unfortunately, the Solr plugin is not 
yet available, and therefore you will have to write your own query 
syntax and parser for this task.


Regards,

[1] http://siren.sindice.com
--
Renaud Delbru

gdeconto wrote:

thx for the reply.

you mean into a multivalue field?  possible, but was wondering if there was
something more flexible than that.  the ability to use a function (ie
myfunction) would open up some possibilities for more complex searching and
search syntax.

I could write my own query parser with special extended syntax, but that is
farther than I wanted to go.



Manepalli, Kalyan wrote:
  

You can copy the dynamic fields value into a different field and query on
that field.

Thanks,
Kalyan Manepalli





  

Re: how to scan dynamic field without specifying each field in query

[gdeconto]

I am thinking that my example was too simple/generic :-U.  It is possible for
more several dynamic fields to exist and other functionality to be required.
i.e. what about if my example had read:

http://localhost:8994/solr/select?q=((Foo1:3 OR Foo2:3 OR Foo3:3 OR …
Foo999:3) AND (Bar1:1 OR Bar2:1 OR Bar3:1...Bar999:1) AND (Etc1:7 OR Etc2:7
OR Etc3:7...Etc:999:7)

obviously a nasty query (and care would be needed for MAX_BOOLEAN_CLAUSES). 
that said, are there other mechanisms to better handle that type of query,
i.e.:

http://localhost:8994/solr/select?q=(myfunction(‘Foo’, 3) AND
myfunction('Bar', 1) AND (myfunction('Etc', 7))


gdeconto wrote:
> 
> say I have a dynamic field called Foo* (where * can be in the hundreds)
> and want to search Foo* for a value of 3 (for example)
> 
> I know I can do this via this:
> 
> http://localhost:8994/solr/select?q=(Foo1:3 OR Foo2:3 OR Foo3:3 OR …
> Foo999:3)
> 
> However, is there a better way?  i.e. is there some way to query by a
> function I create, possibly something like this:
> 
> http://localhost:8994/solr/select?q=myfunction(‘Foo’, 3)
> 
> where myfunction itself iterates thru all the instances of Foo*
> 
> any help appreciated
> 
> 

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25283094.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to scan dynamic field without specifying each field in query

[Aakash Dharmadhikari]

what all other searches you would like to perform on these fields?

>From the proposed function definition I believe that when foo*:3 is to be
searched all foo* would be searched and none are to be excluded. Assuming
that this is the only search that are to be performed on these fields, we
might declare the dynamic field foo* and rather than constructing fields
actual name with property key we can construct it with the "to be searched
value" and store key as the value.

So assume we want to search fooA:X fooB:X fooC:X I would rather store
fooX as the multivalued field and store all A, B, C as its values.

The search query can be FooX:*, that is if field fooX exists, get all the
values for the same.

But again as I asked first, it would depend on what kind of other queries
you want  to perform.

regards,
aakash

2009/9/4 gdeconto 

>
> I am thinking that my example was too simple/generic :-U.  It is possible
> for
> more several dynamic fields to exist and other functionality to be
> required.
> i.e. what about if my example had read:
>
> http://localhost:8994/solr/select?q=((Foo1:3<http://localhost:8994/solr/select?q=%28%28Foo1:3>OR
>  Foo2:3 OR Foo3:3 OR …
> Foo999:3) AND (Bar1:1 OR Bar2:1 OR Bar3:1...Bar999:1) AND (Etc1:7 OR Etc2:7
> OR Etc3:7...Etc:999:7)
>
> obviously a nasty query (and care would be needed for MAX_BOOLEAN_CLAUSES).
> that said, are there other mechanisms to better handle that type of query,
> i.e.:
>
> http://localhost:8994/solr/select?q=(myfunction(<http://localhost:8994/solr/select?q=%28myfunction%28>‘Foo’,
> 3) AND
> myfunction('Bar', 1) AND (myfunction('Etc', 7))
>
>
> gdeconto wrote:
> >
> > say I have a dynamic field called Foo* (where * can be in the hundreds)
> > and want to search Foo* for a value of 3 (for example)
> >
> > I know I can do this via this:
> >
> > http://localhost:8994/solr/select?q=(Foo1:3<http://localhost:8994/solr/select?q=%28Foo1:3>OR
> >  Foo2:3 OR Foo3:3 OR …
> > Foo999:3)
> >
> > However, is there a better way?  i.e. is there some way to query by a
> > function I create, possibly something like this:
> >
> > http://localhost:8994/solr/select?q=myfunction(<http://localhost:8994/solr/select?q=myfunction%28>‘Foo’,
> 3)
> >
> > where myfunction itself iterates thru all the instances of Foo*
> >
> > any help appreciated
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25283094.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: how to scan dynamic field without specifying each field in query

[gdeconto]

I dont have that answer as I was asking a general question, not one for a
specific situation I am encountering).

what I am essentially asking for is: is there a short, simple and generic
method/technique to deal with large numbers of dynamic fields (rather than
having to specify each and every test on each and every dynamic field) in a
query

what originally prompted this question is I was looking at FunctionQueries
(http://wiki.apache.org/solr/FunctionQuery) and started to wonder if there
was some way to create my own functions to handle dynamic fields.


Aakash Dharmadhikari wrote:
> 
> what all other searches you would like to perform on these fields?
> 
> ...
> 

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25297439.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to scan dynamic field without specifying each field in query

[Avlesh Singh]

>
> I dont have that answer as I was asking a general question, not one for a
> specific situation I am encountering).
>
I can understand :)

what I am essentially asking for is: is there a short, simple and generic
> method/technique to deal with large numbers of dynamic fields (rather than
> having to specify each and every test on each and every dynamic field) in a
> query
>
Not as of now. There are a lot of open issues in Solr aiming to handle
dynamic fields in an intuitive way. SolrJ has already been made capable of
binding dynamic field content into Java beans (
https://issues.apache.org/jira/browse/SOLR-1129). Faceting on "myField_*" (
https://issues.apache.org/jira/browse/SOLR-1387) and adding SolrDocuments
with "Map myField_*" (
https://issues.apache.org/jira/browse/SOLR-1357) are just some of the
enhancements on the way.

what originally prompted this question is I was looking at FunctionQueries (
> http://wiki.apache.org/solr/FunctionQuery) and started to wonder if there
> was some way to create my own functions to handle dynamic fields.
>
I don't think you need function queries here. Function queries are supposed
to return score for a document based on their ValueSource. What you probably
need is a custom QueryParser.

Cheers
Avlesh

On Fri, Sep 4, 2009 at 9:48 PM, gdeconto wrote:

>
> I dont have that answer as I was asking a general question, not one for a
> specific situation I am encountering).
>
> what I am essentially asking for is: is there a short, simple and generic
> method/technique to deal with large numbers of dynamic fields (rather than
> having to specify each and every test on each and every dynamic field) in a
> query
>
> what originally prompted this question is I was looking at FunctionQueries
> (http://wiki.apache.org/solr/FunctionQuery) and started to wonder if there
> was some way to create my own functions to handle dynamic fields.
>
>
> Aakash Dharmadhikari wrote:
> >
> > what all other searches you would like to perform on these fields?
> >
> > ...
> >
>
> --
> View this message in context:
> http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25297439.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: how to scan dynamic field without specifying each field in query

[gdeconto]

Thanks Avlesh.

I was only thinking of something 'like' function queries (since they
appeared to have similar behavior).

Agree that custom QueryParser is looking like my only choice.  Now have to
figure out how to do that :-)



Avlesh Singh wrote:
> 
> I don't think you need function queries here. Function queries are
> supposed
> to return score for a document based on their ValueSource. What you
> probably
> need is a custom QueryParser.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/how-to-scan-dynamic-field-without-specifying-each-field-in-query-tp25280228p25300855.html
Sent from the Solr - User mailing list archive at Nabble.com.