Re: loading many documents by ID

2007-02-03 Thread Yonik Seeley
On 2/3/07, Chris Hostetter <[EMAIL PROTECTED]> wrote: the schema creator should still have some say in what kinds of things are allowed/dissalloed though -- the person doing the "update" may not fully understand the underlying model. I think the two concerns should be separated: 1) updateable d

Re: loading many documents by ID

2007-02-02 Thread Chris Hostetter
: I agree. I started down that path, and it gets pretty ugly. I : stopped. I have opted for a syntax that 'updates' all stored fields, : but lets you say explicitly what to do for each field. If there is a : stored field you want to skip, you can specify that in command rather : then in the sc

Re: loading many documents by ID

2007-02-02 Thread Ryan McKinley
> > How about: Iterable Maybe... but that might not be the easiest for request handlers to use... they would then need to spin up a different thread and use a pull model (provide a new doc on demand) rather than push (call addDocument()). With Iterable, you don't need to start a thread to impl

Re: loading many documents by ID

2007-02-02 Thread Yonik Seeley
On 2/1/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: > > Not sure... depends on how update handlers will use it... by update handler, you mean UpdateRequestHandler(s)? or UpdateHandler? Both. > One thing we might not want to get rid of though is streaming > (constructing and adding a document

Re: loading many documents by ID

2007-02-02 Thread Ryan McKinley
1) regardless of the verb (updatable/modifiable) i'm not sure that it makes sense to annotate in the schema the fields that should be copied on update, and not label the feilds that must be "set" on update (ie: the fields that cannot be copied) I agree. I started down that path, and it gets pr

Re: loading many documents by ID

2007-02-01 Thread Chris Hostetter
: 1. Set the "updateable" fields explicitly in the schema. : : : * throw an exception at startup if an updateable field is not stored. : If somewhere down the road we figure out how to efficiently handled : unstored fields, we can remove this error. : * when 'updating', only copy the fields mark

Re: loading many documents by ID

2007-02-01 Thread Ryan McKinley
Not sure... depends on how update handlers will use it... by update handler, you mean UpdateRequestHandler(s)? or UpdateHandler? One thing we might not want to get rid of though is streaming (constructing and adding a document, then discarding it). People are starting to add a lot of documen

Re: loading many documents by ID

2007-02-01 Thread Yonik Seeley
On 2/1/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I am (was?) using DISTINCT to say, only add the unique fields. As implemented, it keeps a Collection for each field name. If the 'mode' is 'DISTINCT' the collection is Set, otherwise List Ah, OK... that does seem useful. How would you feel

Re: loading many documents by ID

2007-02-01 Thread Ryan McKinley
> REPLACE_DOCUMENT > REPLACE_FIELDS > REPLACE_DISTINCT_FIELDS > ADD_FIELDS > ADD_DISTINCT_FIELDS What does "distinct" mean in this context? I am (was?) using DISTINCT to say, only add the unique fields. As implemented, it keeps a Collection for each field name. If the 'mode' is 'DISTINCT' t

Re: loading many documents by ID

2007-02-01 Thread Yonik Seeley
On 2/1/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I have something working that adds a 'mode' to AddUpdateCommand. The modes I need are: Feel free to suggest replacements for the UpdateCommand classes if things become cumbersome. REPLACE_DOCUMENT REPLACE_FIELDS REPLACE_DISTINCT_FIELDS ADD_

Re: loading many documents by ID

2007-02-01 Thread Walter Underwood
On 2/1/07 10:55 AM, "Ryan McKinley" <[EMAIL PROTECTED]> wrote: > > Is there a better word then 'update'? It seems there is already enough > confusion between UpdateHandlers, "Update Plugins", > UpdateRequestHandler etc. Try "modify". Solr uses "update" to include "add". wunder

Re: loading many documents by ID

2007-02-01 Thread Ryan McKinley
What I think I'm seeing is two validation options: 1. Set the "updateable" fields explicitly in the schema. * throw an exception at startup if an updateable field is not stored. If somewhere down the road we figure out how to efficiently handled unstored fields, we can remove this error. * whe

Re: loading many documents by ID

2007-01-31 Thread Erik Hatcher
On Feb 1, 2007, at 12:05 AM, Ryan McKinley wrote: > > We'd have to make it very clear that this only works if all fields are > STORED. Isn't there some way to do this automatically instead of relying on documentation? We might need to add something, maybe a "required" attribute on fields, bu

Re: loading many documents by ID

2007-01-31 Thread Walter Underwood
On 1/31/07 9:05 PM, "Ryan McKinley" <[EMAIL PROTECTED]> wrote: >>> >>> We'd have to make it very clear that this only works if all fields are >>> STORED. >> >> Isn't there some way to do this automatically instead of relying >> on documentation? We might need to add something, maybe a >> "require

Re: loading many documents by ID

2007-01-31 Thread Ryan McKinley
> > We'd have to make it very clear that this only works if all fields are > STORED. Isn't there some way to do this automatically instead of relying on documentation? We might need to add something, maybe a "required" attribute on fields, but a runtime error would be much, much better than a pag

Re: loading many documents by ID

2007-01-31 Thread Walter Underwood
On 1/31/07 3:39 PM, "Chris Hostetter" <[EMAIL PROTECTED]> wrote: > > : Oh, and there have been numerous people interested in "updateable" > : documents, so it would be nice if that part was in the update handler. > > We'd have to make it very clear that this only works if all fields are > STORED.

Re: loading many documents by ID

2007-01-31 Thread Yonik Seeley
On 1/31/07, Erik Hatcher <[EMAIL PROTECTED]> wrote: On Jan 31, 2007, at 6:39 PM, Chris Hostetter wrote: > : Oh, and there have been numerous people interested in "updateable" > : documents, so it would be nice if that part was in the update > handler. > > We'd have to make it very clear that thi

Re: loading many documents by ID

2007-01-31 Thread Erik Hatcher
On Jan 31, 2007, at 6:39 PM, Chris Hostetter wrote: : Oh, and there have been numerous people interested in "updateable" : documents, so it would be nice if that part was in the update handler. We'd have to make it very clear that this only works if all fields are STORED. That is perfectly

Re: loading many documents by ID

2007-01-31 Thread Chris Hostetter
: Oh, and there have been numerous people interested in "updateable" : documents, so it would be nice if that part was in the update handler. We'd have to make it very clear that this only works if all fields are STORED. -Hoss

Re: loading many documents by ID

2007-01-30 Thread Yonik Seeley
On 1/30/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I am working on an SQLUpdatePlugin that needs to set a few fields in the document and leave the rest as they were. Oh, and there have been numerous people interested in "updateable" documents, so it would be nice if that part was in the updat

Re: loading many documents by ID

2007-01-30 Thread Yonik Seeley
On 1/30/07, Ryan McKinley <[EMAIL PROTECTED]> wrote: I am working on an SQLUpdatePlugin that needs to set a few fields in the document and leave the rest as they were. My plan is to load the old documents from the current searcher and overwrite any fields set from SQL. What is the best way to l

loading many documents by ID

2007-01-30 Thread Ryan McKinley
I am working on an SQLUpdatePlugin that needs to set a few fields in the document and leave the rest as they were. My plan is to load the old documents from the current searcher and overwrite any fields set from SQL. What is the best way to load a (potentially) large set of documents by ID? Sho