Good point. I meant the model at the document level: i.e. what
milestones does a document go through in its life cycle. Today:
created --> deleted
With incremental updates:
created --> update1 --> update2 --> deleted
I think what I'm trying to say is that this second threaded sequence
of state
write once, plus the option to the app to keep multiple commit points
around (by customizing the deletion policy).
Actually order of operations / commits very much matters in Lucene today.
Deletions are not idempotent: if you add a doc w/ term X, delete by
term X, add a new doc with term X... tha
Sure. Because of the write once principle. But at some cost
(duplicated data). I was just agreeing that it would not be a good
idea to bake in version-ing by keeping the layers around forever in a
merged index; I wasn't keying in on transactions per se.
Speaking of transactions: I'm not sure if w
On Sat, Apr 3, 2010 at 1:25 AM, Babak Farhang wrote:
>> I think they get merged in by the merger, ideally in the background.
>
> That sounds sensible. (In other words, we wont concern ourselves with
> roll backs--something possible while a "layer" is still around.)
Actually roll backs would still
Grant,
Reading your post a 3rd time, I see my "suggestion" is in fact the
approach you describe. Sorry for being redundant.
-Babak
On Fri, Apr 2, 2010 at 11:25 PM, Babak Farhang wrote:
>> I think they get merged in by the merger, ideally in the background.
>
> That sounds sensible. (In other w
> I think they get merged in by the merger, ideally in the background.
That sounds sensible. (In other words, we wont concern ourselves with
roll backs--something possible while a "layer" is still around.)
I've been thinking about this problem also. One approach discussed
earlier in these mailing
On Apr 2, 2010, at 2:50 AM, Babak Farhang wrote:
> [Late to this party, but thought I'd chime in]
>
> I think this "layer" concept is right on. But I'm wondering about the
> life cycle of these layers. Do layers live forever? Or do they
> collapse at some point? (Like, as I think was already p
[Late to this party, but thought I'd chime in]
I think this "layer" concept is right on. But I'm wondering about the
life cycle of these layers. Do layers live forever? Or do they
collapse at some point? (Like, as I think was already pointed out,
deletes are when segments are merged today.)
-Ba
of URLs, database keys or filenames which can
> be used. Also, b) is not too hard to handle in app code if you always use the
> IndexWriter.updateDocument(term,doc) method for inserts.
>
>
> Cheers,
> Mark
>
> From: Grant Ingersoll
> To: java-dev@lucene.apache.org
> Sent: Mon, 29 March, 201
r.updateDocument(term,doc) method for inserts.
Cheers,
Mark
From: Grant Ingersoll
To: java-dev@lucene.apache.org
Sent: Mon, 29 March, 2010 13:11:56
Subject: Re: Incremental Field Updates
On Mar 29, 2010, at 2:26 AM, Mark Harwood wrote:
>
>
>>
&g
On 2010-03-29 15:11, Uwe Goetzke wrote:
The filed this as patent, too:
http://www.freepatentsonline.com/y2009/0228528.html
.. which is not granted yet, right? It's a patent application. Besides,
I live in EU ;)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __
The filed this as patent, too:
http://www.freepatentsonline.com/y2009/0228528.html
Regards
Uwe Goetzke
-Ursprüngliche Nachricht-
Von: Andrzej Bialecki [mailto:a...@getopt.org]
Gesendet: Montag, 29. März 2010 14:50
An: java-dev@lucene.apache.org
Betreff: Re: Incremental Field Updates
On 2010-03-29 12:26, Michael McCandless wrote:
I agree this is a long overdue feature... we need to get it into
Lucene somehow.
I like the Layers analogy... I think that will work well with Lucene's
transactional semantics, ie a prior commit point would continue to see
the index before the updat
On Mar 29, 2010, at 2:26 AM, Mark Harwood wrote:
>
>>
>>> Of course introducing the idea of updates also introduces the notion of a
>>> primary key and there's probably an entirely separate discussion to be had
>>> around user-supplied vs Lucene-generated keys.
>>
>> Not sure I see that need
>>Who ever said that some_condition should point to a unique document?
>
> My assumption was, for now, we were still talking about the simpler case of
> updating a single document.
> If we extend the discussion to support set-based updates it's worth
> considering the common requirements for upda
March, 2010 11:05:39
Subject: Re: Incremental Field Updates
>>Variant d) sounds most logical? And enables all sorts of fun stuff.
>
> So the duplicate-key docs can have different values for initial-insert fields
> but partial updates will cause sharing of a common field value?
&
I agree this is a long overdue feature... we need to get it into
Lucene somehow.
I like the Layers analogy... I think that will work well with Lucene's
transactional semantics, ie a prior commit point would continue to see
the index before the updates but new commit points would see the
updates.
>>Variant d) sounds most logical? And enables all sorts of fun stuff.
>
> So the duplicate-key docs can have different values for initial-insert fields
> but partial updates will cause sharing of a common field value?
> And subsequent same-key doc inserts do or don't share these previous
> "part
"partial-update" values?
Sounds like a complex model for users to understand let alone code support for.
Everyone gets primary keys though.
- Original Message
From: Earwin Burrfoot
To: java-dev@lucene.apache.org
Sent: Mon, 29 March, 2010 10:14:24
Subject: Re: Incremental Field Upd
>>If someone needs this, it can be built over lucene, without
>>introducing it as a core feature and needlessly complicating things.
>
> I think with any partial-update feature the *absence* of primary key support
> would "needlessly complicate things":
> If Lucene is not capable of performing du
>I can delete by lucene-generated docId.
Which users used to have to find by first coding a primary-key-term search.
Delete by term removed this step to make life easier.
>If someone needs this, it can be built over lucene, without
>introducing it as a core feature and needlessly complicating
> Of course introducing the idea of updates also introduces the notion of a
> primary key and there's probably an entirely separate discussion to be had
> around user-supplied vs Lucene-generated keys.
Not sure I see that need. Can you explain your reasoning a bit more?
>>> If you
On 29 Mar 2010, at 07:45, Earwin Burrfoot wrote:
Of course introducing the idea of updates also introduces the notion of a
primary key and there's probably an entirely separate discussion to be had
around user-supplied vs Lucene-generated keys.
Not sure I see that need. Can you explain your re
>>> Of course introducing the idea of updates also introduces the notion of a
>>> primary key and there's probably an entirely separate discussion to be had
>>> around user-supplied vs Lucene-generated keys.
>> Not sure I see that need. Can you explain your reasoning a bit more?
> If you want to u
Of course introducing the idea of updates also introduces the notion of a
primary key and there's probably an entirely separate discussion to be had
around user-supplied vs Lucene-generated keys.
Not sure I see that need. Can you explain your reasoning a bit more?
If you want to update a do
On Mar 27, 2010, at 11:14 AM, Mark Harwood wrote:
> Of course introducing the idea of updates also introduces the notion of a
> primary key and there's probably an entirely separate discussion to be had
> around user-supplied vs Lucene-generated keys.
Not sure I see that need. Can you explain
Of course introducing the idea of updates also introduces the notion of a
primary key and there's probably an entirely separate discussion to be had
around user-supplied vs Lucene-generated keys.
That aside, the biggest concern for me here is the impact that this is likely
to have on search -
First off, this is something I've had in my head for a long time, but don't
have any code.
As many of you know, one of the main things that vexes any search engine based
on an inverted index is how to do fast updates of just one field w/o having to
delete and re-add the whole document like we d
28 matches
Mail list logo
