Grant Ingersoll wrote:
On Aug 10, 2009, at 6:28 PM, Mark Miller wrote:
Grant Ingersoll wrote:
On Aug 10, 2009, at 5:12 PM, Shai Erera wrote:
Maybe we should follow what I seem to read from Earwin and Grant -
come up w/ real use cases, try to implement them w/ the current
API, then if it's impossible, discuss how we can make the current
API more adaptive. If at the end of this we'll get back to the new
API, then we'll at least feel better about it, and more convinced
it is the way to go.
Well, I have real use cases for it, but all of it is still missing
the biggest piece: search side support. It's the 900 lb. elephant
in the room. The 500 lb. elephant is the fact that all these
attributes, AIUI, require you to hook in your own indexing chain,
etc. in order to even be indexed, which is all package private
stuff. It's not even clear to me what happens right now if you
were to, say have a Token Stream that, say, had only one Attribute
on it and none of the existing attributes (term buffer, length,
position, etc.) Please correct me if I am wrong, I still don't have
a deep understanding of it all.
Michael has always been up front that this new API is in preparation
for flexible indexing. It doesn't give us the goodness - he has laid
out the reasons for moving before the goodness comes more than once I
think. From my understanding, Michael looked at what Mike was doing
in one of his flexible indexing patches, wondered how some of the
TokenStream stuff was going to work well with it, and came up with
this new API as a solution. Yes - it gets us nothing now. But its a
big move, and there is no need to do everything at once - in fact it
would probably be harder to do it all at once - the rest has always
been on the table. 3.0 has always been convenient to push it before,
as deprecations can than be removed. Nothing forcing us to make that
decision now though.
Honestly, though, it really gives you very little over the current,
well functioning payloads capability other than stronger typing, the
ability to pick only those attributes that you want indexed (in
theory) and a byte (or so) of savings per any token that has a
payload, and we _HAVE_ right now, search support for payloads.
Payloads gives us nothing as developers - you can't use that
functionality without taking it from the users - payloads are for users.
Flexible indexing will lead to all kinds of little cool things - the
likes of which have been discussed a lot in older emails. It will
likely lead to things we cannot predict as well. Everything will be
more flexible. It also could play a part in CSF, and work on allowing
custom files to plug into merging. Plus everything else thats been
mentioned (pfor, etc) I've been sold on the long term benefits. I
don't think you need these API for them, but its my understanding it
helps solve part of the equation.
A bunch of issues have come up. To my knowledge, they have been
addressed with vigor every time. If someone is unhappy with how
something has been addressed, and it needs to be addressed further,
please speak up.
Um, that's what I've been doing. Vigor is good. I very much
appreciate everyone's work. From what I can tell, most devs here are
unsure at best what to do with their existing Analyzer capabilities.
I've actually implemented a couple of new TokenFilter's using the new
APIs. I like that aspect of it. I'm just not sure on the back compat
hoops (and yes, I asked for them). But I'm also operating under the
assumption that our BC approach isn't going to change anytime soon,
such that it is very important that these new capabilities are worked
out (and I don't just mean little performance nicks here and there, I
mean in terms of usability and performance).
I'm not just responding to just you there, but more to the growing pack
of those speaking against the new API. I don't see specific issues being
brought up - the only issues I have seen brought up have been addressed
in JIRA issues that have received no comments indicating the fix was not
good enough. So we are seeing a lot of general complaints, but specific
complaints have been addressed as far as I can tell.
As far as back compat - is it really still considered an issue? We have
broken back compat in this release wherever it was convenient to do so.
I suspect that will continue. I just wish our policy reflected how
things actually work (and I think they work as they should, based on the
circumstances that lead to each decision).
Let's put it this way: We expect to release 2.9 within the month
(which is very short in Lucene time). That will give us a sum total
of, what, 2.5 weeks of review by devs for some very major changes? I
want 3.0 as much as anyone (I've been pushing for 1.5 support for at
least 2 years now), but I don't want us to be in a hole going into it
because we felt rushed right when the "finish" was so close.
Otherwise, I don't think the sky is falling - I think the new API is
being shaken out.
I agree its not falling. It never is. This is in fact how the
process works. People are doing the right thing here by discussing it
and working on it.
Thats kind of in response to the ground swell that appeared to be
building to roll back or hold off on the new API. To me, we would do
that if the sky was falling. As long as specific issues are being
addressed (and the number issues has not been that high), I just don't
see a reason to hold off on the current plan.
Oh, and now it seems the new QP is dependent on it all.
Dependent how?
Attribute and a whole slew of AttributeImpls.
Oh, because it uses the Attributes. I think the new QueryParser is its
own kettle of fish. It really shouldn't have a back compat promise while
it lives in contrib. It needs to be shaked out before it could possibly
replace the current parser.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
