Hoss'ss time crunch -or- why i haven't commented on patches lately.
Just an FYI: you may have noticed i haven't been as on the ball with reviewing patches hte past few weeks ... i've been house hunting and last week I found a good one. i just started escrow and the whole process is sucking up a lot of time -- it doesn't help that i've lived in the same apartment for about 10 years now, and I don't even remember what "moving" is like. between my agent, my mortgage broker, the inspection, the insurance, the title company, etc... i'm barely managing to cover my day job, so at night i'm just trying to keep up with reading/replying to mail. I'm not avoiding any patches ... well that's not true, i'm avoiding all of them -- but not because i don't want to look at them; it's just that reading/writing email is easier then reading/analyzing code. With any luck, in 3 weeks I'll have a lot of free time to review/apply patches while sitting in my new yard. -Hoss
Re: svn commit: r519107 - in /lucene/solr/trunk: CHANGES.txt example/exampledocs/spellchecher.xml example/solr/conf/solrconfig.xml lib/lucene-spellchecker-2.2-dev.jar src/java/org/apache/solr/request/
Should I really move the SpellCheckerRequestHandler to o.a.solr.handler instead? I see request handlers in both handler and request packages. There are two handlers in o.a.s.request: DisMaxRequestHandler.java StandardRequestHandler.java all the other handlers are in o.a.s.handler. Moving DisMax and Standard is harder because people may already be extending them. (At some point we may want deprecate those and move them to .handler) As we add things like CSVUpdateRequestHandler, LukeRequestHandler, SQLUpdateReqeustHandler it seems cleaner (to me) to keep them isolated from the response writers, parameter parsing etc...
[jira] Resolved: (SOLR-58) Change Admin components to return XML like the rest of the system
[ https://issues.apache.org/jira/browse/SOLR-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic resolved SOLR-58. -- Resolution: Fixed I'll resolve this as Fixed, even though it should be resolved as "No longer relevant", since I believe Ryan has changed or will change all admin stuff to work with handlers/writersor at least that's what I recall reading. > Change Admin components to return XML like the rest of the system > - > > Key: SOLR-58 > URL: https://issues.apache.org/jira/browse/SOLR-58 > Project: Solr > Issue Type: New Feature > Components: web gui >Reporter: Otis Gospodnetic > Assigned To: Otis Gospodnetic >Priority: Minor > Attachments: SOLR-58.patch > > > I need to expose the admin functionality to an external application. I think > returning admin data as XML may be a good and simple first step towards that. > To do that I think I'll mostly need to modify JSPs (but I haven't had a good > look at Admin GUI yet). From what I saw a few weeks ago when I briefly > looked at this, no Java code will need to be modified. If you have concrete > ideas about how this should be done, please comment before I start next week > (week of October 23rd 2006). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: svn commit: r519107 - in /lucene/solr/trunk: CHANGES.txt example/exampledocs/spellchecher.xml example/solr/conf/solrconfig.xml lib/lucene-spellchecker-2.2-dev.jar src/java/org/apache/solr/request/
Thanks for comments, Ryan. Took care of 2, 3, and 4. Should I really move the SpellCheckerRequestHandler to o.a.solr.handler instead? I see request handlers in both handler and request packages. Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Ryan McKinley <[EMAIL PROTECTED]> To: solr-dev@lucene.apache.org Sent: Sunday, March 18, 2007 8:41:20 PM Subject: Re: svn commit: r519107 - in /lucene/solr/trunk: CHANGES.txt example/exampledocs/spellchecher.xml example/solr/conf/solrconfig.xml lib/lucene-spellchecker-2.2-dev.jar src/java/org/apache/solr/request/SpellCheckerRequestHandler.java A few trivial comments on SpellCheckerRequestHandler: 1. Should it go in "org.apache.solr.handler" rather then "org.apache.solr.request"? As the number of special handlers increases, it seems good to have them semi-isolated. 2. replace tabs with two spaces 3. The file needs the svn props set 4. getVersion() should return "$Revision: 501512 $"; rather then SolrCore.version nit pick nit pick nit pick. otherwise looks good!
[jira] Updated: (SOLR-162) lucene index browser / admin helpers (Luke)
[
https://issues.apache.org/jira/browse/SOLR-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McKinley updated SOLR-162:
---
Attachment: SOLR-162-Admin-XML-luke.patch
Updated to:
* use SOLR-182 rather then implement its own dynamic loading at startup
* apply without conflicts
* use Luke 0.7 style to represent more field properties:
"key":{
"I":"Indexed",
"T":"Tokenized",
"S":"Stored",
"M":"Multivalued",
"V":"TermVector Stored",
"o":"Store Offset With TermVector",
"p":"Store Position With TermVector",
"O":"Omit Norms",
"L":"Lazy",
"B":"Binary",
"C":"Compressed",
"f":"Sort Missing First",
"l":"Sort Missing Last"},
"fields":{
"id":{
"type":"string",
"schema":"I-SOl",
"flags":"I-SO-",
...
> lucene index browser / admin helpers (Luke)
> ---
>
> Key: SOLR-162
> URL: https://issues.apache.org/jira/browse/SOLR-162
> Project: Solr
> Issue Type: New Feature
> Components: web gui
>Reporter: Ryan McKinley
>Priority: Minor
> Attachments: SOLR-162-Admin-XML-luke.patch,
> SOLR-162-Admin-XML-luke.patch, SOLR-162-Admin-XML-luke.patch,
> SOLR-162-Admin-XML-luke.patch, SOLR-162-Admin-XML-luke.patch
>
>
> Luke (http://www.getopt.org/luke/) is a great tool to help learn / understand
> / debug lucene indexes.
> Solr already does a lot of what luke does... but it could do a bit more.
> Specifically:
> * browse top terms across all fields (similar to faceting)
> * browse lucene documents / properties directly
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-193) General SolrDocument interface to manage field values.
[
https://issues.apache.org/jira/browse/SOLR-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McKinley updated SOLR-193:
---
Attachment: SOLR-193-SolrDocument.patch
yes:
Object toObject( Fieldable );
is better.
> General SolrDocument interface to manage field values.
> --
>
> Key: SOLR-193
> URL: https://issues.apache.org/jira/browse/SOLR-193
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan McKinley
> Attachments: SOLR-193-SolrDocument.patch,
> SOLR-193-SolrDocument.patch, SOLR-193-SolrDocument.patch
>
>
> In an effort to make SOLR-139 (the "modify" command) more manageable, i
> extracted out a large chunk. This patch adds a general SolrDocument
> interface and includes a concrete implementation (SimpleSolrDoc)
> SOLR-139 needs some way to transport document values independent of the
> lucene Document. This is required for the INCREMENT command and useful for
> modifying documents. SolrDocument is also generally useful for SOLR-20
> - - - - - -
> The one (potentially) controversial part is that I added a function to
> FieldType:
> public Object toExternalValue(Fieldable f);
> This asks each field type to convert its Fieldable into its real type, for
> example IntField.java has:
> public Integer toExternalValue(Fieldable f) {
>return Integer.valueOf( toExternal(f) );
> }
> By default, it returns a string value. If this addition is too much, there
> are other (less clean) ways to handle the INCREMENT command. My real
> motivation for this addition is that it makes it possible to implement an
> embeddable SOLR-20 client that does not need an HTTP connection.
> - - - -
> The SimpleSolrDoc implementation was written for SOLR-20. It needs to play
> nice with EL, so it implements a few extra map function that may not seem
> necessary:
> ${doc.values['name']]} gets a collection
> ${doc.valueMap['name']]} gets a single value for the field
> - - - -
> The tests cover all "toExternalValue" changes in schema.*
> SimpleSolrDoc and DocumentBuilder have 100% test coverage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-176) Add detailed timing data to query response output
[ https://issues.apache.org/jira/browse/SOLR-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482550 ] Hoss Man commented on SOLR-176: --- i don't have any strong opinions, but i agree with your plan about putting some warnings in the RTTimer javadocs. Random thought: it might be cleaner/easier in the long run to create/register a hog Handler that records timing differences and add calls to Logger.entering and Logger.exiting on the various methods. someone may have already implemented such a Handler > Add detailed timing data to query response output > - > > Key: SOLR-176 > URL: https://issues.apache.org/jira/browse/SOLR-176 > Project: Solr > Issue Type: New Feature > Components: search >Affects Versions: 1.2 >Reporter: Mike Klaas > Assigned To: Mike Klaas >Priority: Minor > Fix For: 1.2 > > Attachments: dtiming.patch, dtiming.patch > > > see > http://www.nabble.com/%27accumulate%27-copyField-for-faceting-tf3329986.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] LUCENE-831 and FieldCache warming
: Sounds interesting. I am not sure I quite understand what this patch does, : but if it helps improve the first sort after a commit that would rock :) the main issue is that Solr currently has no visibility into much of hte caching done by the underlying Lucene implementation, so it can't "autowarm" anything ... you as a Solr instance owner can explicitly warm fields you know you sort on, or fields you know you facet on -- but that requires keeping that config upto date ... adding an API like this to Lucene would help make it possible for Solr to do this automaticaly based on simpler config (ie; declaring an autowarmCount on the new fieldCache) : results so we can show newest articles/posts first, but currently cannot, : because of relatively high volume of inserts/commits(15000 new posts pr day : on our forum). as i said, if you just want to support sorting on a few fields, you can configure explicit warming queries that do that sorting (and populate those caches) in your solrconfig.xml -Hoss
[jira] Updated: (SOLR-183) add getRequiredParameter() to SolrParams
[
https://issues.apache.org/jira/browse/SOLR-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McKinley updated SOLR-183:
---
Attachment: SOLR-183-required-param.patch
Looks good. thanks. I agree it is cleaner as a decorator. As a decorator, I
think getInt( 'xxx', defaultVal ) shoould work, not throw an exception.
I don't follow the strict/not strict logic to getFieldParam... If you don't
want strict checking, use the normal SolrParams, if you do, use
RequiredSolrParams
This update changes things so the basic contract with RequiredSolrParams is
that you get back a valid non-null value (unless you pass it in as a default)
* functions with default values call the wrapped params directly
* replaced tabs with 2 spaces
* removed the 'strict' field logic
> add getRequiredParameter() to SolrParams
>
>
> Key: SOLR-183
> URL: https://issues.apache.org/jira/browse/SOLR-183
> Project: Solr
> Issue Type: Wish
>Reporter: Ryan McKinley
>Priority: Trivial
> Attachments: RequiredSolrParams.java, SOLR-183-required-param.patch,
> SOLR-183-required-param.patch, SOLR-183-required-param.patch,
> SOLR-183-required-param.patch, SOLR-183-required-param.patch
>
>
> I find myself including this with every patch, so i'll just separate it out.
> This simply adds a utilty function to SolrParams that throws a 400 if the
> parameter is missing:
> /** returns the value of the param, or throws a 400 exception if missing */
> public String getRequiredParameter(String param) throws SolrException {
> String val = get(param);
> if( val == null ) {
> throw new SolrException( 400, "Missing parameter: "+param );
> }
> return val;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-183) add getRequiredParameter() to SolrParams
[
https://issues.apache.org/jira/browse/SOLR-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.J. Larrea updated SOLR-183:
-
Attachment: SOLR-183-required-param.patch
> add getRequiredParameter() to SolrParams
>
>
> Key: SOLR-183
> URL: https://issues.apache.org/jira/browse/SOLR-183
> Project: Solr
> Issue Type: Wish
>Reporter: Ryan McKinley
>Priority: Trivial
> Attachments: RequiredSolrParams.java, SOLR-183-required-param.patch,
> SOLR-183-required-param.patch, SOLR-183-required-param.patch,
> SOLR-183-required-param.patch
>
>
> I find myself including this with every patch, so i'll just separate it out.
> This simply adds a utilty function to SolrParams that throws a 400 if the
> parameter is missing:
> /** returns the value of the param, or throws a 400 exception if missing */
> public String getRequiredParameter(String param) throws SolrException {
> String val = get(param);
> if( val == null ) {
> throw new SolrException( 400, "Missing parameter: "+param );
> }
> return val;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-183) add getRequiredParameter() to SolrParams
[
https://issues.apache.org/jira/browse/SOLR-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482455
]
J.J. Larrea commented on SOLR-183:
--
I totally agree with Ryan that the question I raised about the value of
specifying required params in solrconfig.xml RH definitions should be separated
from this simpler programmer-API case. I will speak no more of that on
SOLR-183.
Ryan, after looking at your patch #4 I've had a change of heart about the
getRequiredXXX approach. To do it properly would require reduplication of
every method signature, e.g. getFieldInt() and so forth, and wouldn't make any
use of the bottleneck imposed by get/getParams. Hoss' decorator approach
coupled with your improved error handling automagically makes everything work
with a trivial subclass.
This time I implemented and tested everything (attachment #5).
RequiredSolrParams is kept as a freestanding class which can be externally
instantiated, but is also returned by a SolrParams.required() convenience
method so we could stash a reference if desired, e.g.
params.required().getInt("xxx")
params.required().getBoolean("yyy")
(but the wasted cycles and amount of garbage created from allocating a new one
is pretty trivial, so perhaps it's best not to add a slot to SolrParams)
In the bottleneck approach the inline-default methods e.g. getBool("xxx", true)
will fail when called on requires - but I think that is not such a bad thing.
Could be fixed if so desired with a _get().
One open question is getFieldParam: Should the semantics of
required.getFieldParam("facet.limit", "abc") be to fail if the parameter is not
supplied for the field (e.g. f.abc.facet.limit), or not supplied for either the
field or as a general default (e.g. facet.limit)? In the former case we don't
need to override getFieldParam. I can't think of a reason that one would want
to require explicit field values and disallow general values, but perhaps
someone else could, and a 'field strictness" flag should be supplied in the
RequiredSolrParams constructor. For the moment I made it non-strict, but put
in a public value allowing that to be controlled.
I changed the order of operations in SolrParamTest so it starts at the simplest
cases (present and non-required and inline defaults), then malformed values,
then required values. I added the fall-through case for getFieldXXX. I also
started some tests of DefaultSolrParams, to be extended to to
AppendedSolrParams (getParams needs testing as well).
> add getRequiredParameter() to SolrParams
>
>
> Key: SOLR-183
> URL: https://issues.apache.org/jira/browse/SOLR-183
> Project: Solr
> Issue Type: Wish
>Reporter: Ryan McKinley
>Priority: Trivial
> Attachments: RequiredSolrParams.java, SOLR-183-required-param.patch,
> SOLR-183-required-param.patch, SOLR-183-required-param.patch
>
>
> I find myself including this with every patch, so i'll just separate it out.
> This simply adds a utilty function to SolrParams that throws a 400 if the
> parameter is missing:
> /** returns the value of the param, or throws a 400 exception if missing */
> public String getRequiredParameter(String param) throws SolrException {
> String val = get(param);
> if( val == null ) {
> throw new SolrException( 400, "Missing parameter: "+param );
> }
> return val;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-193) General SolrDocument interface to manage field values.
[
https://issues.apache.org/jira/browse/SOLR-193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482438
]
Yonik Seeley commented on SOLR-193:
---
> toExternalValue(Fieldable f)
"external" value sort of already means the human-readable serialized textual
representation.
What about something like toObject() which would return the appropriate Java
object (such as Integer, Long, Date, etc)?
> General SolrDocument interface to manage field values.
> --
>
> Key: SOLR-193
> URL: https://issues.apache.org/jira/browse/SOLR-193
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan McKinley
> Attachments: SOLR-193-SolrDocument.patch, SOLR-193-SolrDocument.patch
>
>
> In an effort to make SOLR-139 (the "modify" command) more manageable, i
> extracted out a large chunk. This patch adds a general SolrDocument
> interface and includes a concrete implementation (SimpleSolrDoc)
> SOLR-139 needs some way to transport document values independent of the
> lucene Document. This is required for the INCREMENT command and useful for
> modifying documents. SolrDocument is also generally useful for SOLR-20
> - - - - - -
> The one (potentially) controversial part is that I added a function to
> FieldType:
> public Object toExternalValue(Fieldable f);
> This asks each field type to convert its Fieldable into its real type, for
> example IntField.java has:
> public Integer toExternalValue(Fieldable f) {
>return Integer.valueOf( toExternal(f) );
> }
> By default, it returns a string value. If this addition is too much, there
> are other (less clean) ways to handle the INCREMENT command. My real
> motivation for this addition is that it makes it possible to implement an
> embeddable SOLR-20 client that does not need an HTTP connection.
> - - - -
> The SimpleSolrDoc implementation was written for SOLR-20. It needs to play
> nice with EL, so it implements a few extra map function that may not seem
> necessary:
> ${doc.values['name']]} gets a collection
> ${doc.valueMap['name']]} gets a single value for the field
> - - - -
> The tests cover all "toExternalValue" changes in schema.*
> SimpleSolrDoc and DocumentBuilder have 100% test coverage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-124) use NewIndexModifier, LUCENE-565
[ https://issues.apache.org/jira/browse/SOLR-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482414 ] Yonik Seeley commented on SOLR-124: --- > Does anyone have a feeling on whether LUCENE-565 will improve performance? Probably not much, if at all. Deletions happen after a segment flush, which is slightly less efficient (more indexreaders that need to be opened), but does get deletions in the index faster (meaning they are more likely to be "squeezed" out by a subsequent segment merge). One small advantage to LUCENE-565 is that overwriting is atomic... you can't crash and see duplicates. The patch has been changed around multiple times, and it would now be necessary to include a lucene package in solr to get access to package-protected stuff that would allow efficient delete-by-query. > use NewIndexModifier, LUCENE-565 > > > Key: SOLR-124 > URL: https://issues.apache.org/jira/browse/SOLR-124 > Project: Solr > Issue Type: Improvement > Components: update >Reporter: Yonik Seeley > > LUCENE-565 adds extension points to the IndexWriter, and adds delete-by-term > functionality. > We should probably take advantage of this (when available) in our > UpdateHandler (a new one, or modify DU2?) > and perhaps implement a more efficient deleteByQuery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-139) Support updateable/modifiable documents
[ https://issues.apache.org/jira/browse/SOLR-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan McKinley updated SOLR-139: --- Attachment: SOLR-139-IndexDocumentCommand.patch depending on SOLR-193 > Support updateable/modifiable documents > --- > > Key: SOLR-139 > URL: https://issues.apache.org/jira/browse/SOLR-139 > Project: Solr > Issue Type: Improvement > Components: update >Reporter: Ryan McKinley > Attachments: SOLR-139-IndexDocumentCommand.patch, > SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, > SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, > SOLR-139-IndexDocumentCommand.patch, SOLR-139-IndexDocumentCommand.patch, > SOLR-139-XmlUpdater.patch > > > It would be nice to be able to update some fields on a document without > having to insert the entire document. > Given the way lucene is structured, (for now) one can only modify stored > fields. > While we are at it, we can support incrementing an existing value - I think > this only makes sense for numbers. > for background, see: > http://www.nabble.com/loading-many-documents-by-ID-tf3145666.html#a8722293 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-193) General SolrDocument interface to manage field values.
[
https://issues.apache.org/jira/browse/SOLR-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McKinley updated SOLR-193:
---
Attachment: SOLR-193-SolrDocument.patch
> General SolrDocument interface to manage field values.
> --
>
> Key: SOLR-193
> URL: https://issues.apache.org/jira/browse/SOLR-193
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan McKinley
> Attachments: SOLR-193-SolrDocument.patch, SOLR-193-SolrDocument.patch
>
>
> In an effort to make SOLR-139 (the "modify" command) more manageable, i
> extracted out a large chunk. This patch adds a general SolrDocument
> interface and includes a concrete implementation (SimpleSolrDoc)
> SOLR-139 needs some way to transport document values independent of the
> lucene Document. This is required for the INCREMENT command and useful for
> modifying documents. SolrDocument is also generally useful for SOLR-20
> - - - - - -
> The one (potentially) controversial part is that I added a function to
> FieldType:
> public Object toExternalValue(Fieldable f);
> This asks each field type to convert its Fieldable into its real type, for
> example IntField.java has:
> public Integer toExternalValue(Fieldable f) {
>return Integer.valueOf( toExternal(f) );
> }
> By default, it returns a string value. If this addition is too much, there
> are other (less clean) ways to handle the INCREMENT command. My real
> motivation for this addition is that it makes it possible to implement an
> embeddable SOLR-20 client that does not need an HTTP connection.
> - - - -
> The SimpleSolrDoc implementation was written for SOLR-20. It needs to play
> nice with EL, so it implements a few extra map function that may not seem
> necessary:
> ${doc.values['name']]} gets a collection
> ${doc.valueMap['name']]} gets a single value for the field
> - - - -
> The tests cover all "toExternalValue" changes in schema.*
> SimpleSolrDoc and DocumentBuilder have 100% test coverage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] LUCENE-831 and FieldCache warming
Chris Hostetter wrote: > > > i just wanted to draw peoples attention to a Lucene-Java patch i've been > experimenting with to change the way the internal FieldCaches work... > > http://issues.apache.org/jira/browse/LUCENE-831 > > ...this might be of special itnerest to Solr developers because it would > let us change the low levels of the FieldCache (used by sorting and > faceting) so that it could be backed by a SolrCache and autoWarmed when > new IndexReaders are opened. (this isn't in the patch) > > Sounds interesting. I am not sure I quite understand what this patch does, but if it helps improve the first sort after a commit that would rock :) We have a number of services where we would like to support sorting search results so we can show newest articles/posts first, but currently cannot, because of relatively high volume of inserts/commits(15000 new posts pr day on our forum). -- View this message in context: http://www.nabble.com/LUCENE-831-and-FieldCache-warming-tf3416220.html#a9568054 Sent from the Solr - Dev mailing list archive at Nabble.com.
