It would seem the next step would be, as Robert suggests, to make a transaction a first class citizen. The XA protocol will require Lucene to communicate with the TM about what transactions it has completed to help in failure recovery and transaction management. I can certainly see the need for a better transaction abstraction to help with this.
A little enlightenment on this would be great robert. I am very interested in it for future projects.
And I have to point out...it just seems logical that we would make things so that the index was consistent at some point before taking the next step of making it consistent with other resources...no? I am just still confused about Roberts objections to what is going on here. I think that it would be a real leap forward to get it done though.
Also, as he mentioned, we really need a good distributed system that allows for index partitioning. Thats the ticket to more enterprise adoption. Could be Solr's work though...
Michael McCandless wrote:
Robert, besides LUCENE-1044 (syncing on commit), what is the Lucene core missing in order for you (or, someone) to build XA compliance on top of it? Ie, you can open a writer with autoCommit=false and no changes are committed until you close it. You can abort the session by calling writer.abort(). What's still missing, besides LUCENE-1044? Mike robert engels wrote:One more example on this. A lot of work was done on transaction support. I would argue that this falls way short of what is needed, since there is no XA transaction support. Since the lucene index (unless stored in an XA db) is a separate resource, it really needs XA support in order to be consistent with the other resources.All of the transaction work that has been performed only guarantees that barring a physical hardware failure the lucene index can be opened and used at a known state. This index though is probably not consistent with the other resources.All that was done is that we can now guarantee that the index is consistent at SOME point in time.Given the work that was done, we are probably closer to adding XA support, but I think this would be much easier if the concept of a transaction was made first class through the API (and then XA transactions need to be supported).On Jan 22, 2008, at 2:49 PM, robert engels wrote:I don't think group C is interested in bug fixes. I just don't see how Lucene is at all useful if the users are encountering any bug - so they either don't use that feature, or they have already developed a work-around (or they have patched the code in a way that avoids the bug, yet is specific to their environment).For example, I think the NFS work (bugs, fixes, etc.) was quite substantial. I think the actual number of people trying to use NFS is probably very low - as the initial implementation had so many problems (and IMO is not a very good solution for distributed indexes anyway). So all the work in trying to make NFS work "correctly" behind the scenes may have been inefficient, since a more direct, yet major fix may have solved the problem better (like distributed server support, not shared index access).I just think that trying to maintain API compatibility through major releases is a bad idea. Leads to bloat, and complex code - both internal and external. In order to achieve great gains in usability and/or performance in a mature product like Lucene almost certainly requires massive changes to the processes, algorithms and structures, and the API should change as well to reflect this.On Jan 22, 2008, at 2:30 PM, Chris Hostetter wrote:: If they are " no longer actively developing the portion of the code that's : broken, aren't seeking the new feature, etc", and they stay back on old : versions... isn't that exactly what we want? They can stay on the old version,: and new application development uses the newer version. This basically mirrors a philosophy that is rising in the Perl community evangelized by (a really smart dude named chromatic) ..."why are we worry about the effect of upgrades on users who don't upgrade?"The problem is not all users are created equal and not all users upgradefor the same reasons or at the same time... Group A: If someone is paranoid about upgrading, and is still runninglucene1.4.3 because they are afraid if they upgrade their app will break and they don't want to deal with it; they don't care about known bugs inlucene1.4.3, as long as those bugs haven't impacted them yet -- these people aren't going to care wether we add a bunch of new methods tointerfaces, or remove a bunch of public methods from arbitrary releases, because they are never going to see them. They might do a total rewrite of their project later, and they'll worry about it then (when they havelots of time and QA resources)Group: B: At the other extreme, are the "free-spirited" developers (god i hate that that the word "agile" has been co-opted) who are always eager toupgrade to get the latest bells and whistles, and don't mind makingchanges to code and recompiling everytime they upgrades -- just as long asthere are some decent docs on what to change.Croup: C: In the middle is a larg group of people who are interested inupgrading, who want bug fixes, are willing to write new code to take advantage of new features, in some cases are even willing to make small or medium changes their code to get really good performanceimprovements ... but they don't have a lot of time or energy to constantly rewrite big chunks of their app. For these people, knowing that they can "drop in" the new version and it will work is a big reason why there arewilling to upgrade, and why they are willing to spend soem time tweaking code to take advantage of the new features and the newperformacne enhaced APIs -- becuase they don't have to spend a lot of timejust to get the app working as well as it was before. To draw an analogy...Group A will stand in one place for a really long time no matter how easythe path is. Once in a great while they will decide to march forward dozens of miles in one big push, but only once they feel they have adequate resources to make the entire trip at once. Group B likes to frolic, and will happily take two sptens backward and then 3 steps forward every day.Group C will walk forward with you at a steady pace, and occasionally even take a step back before moving forward, but only if the path is clear andnot very steap.: I bet, if you did a poll of all Lucene users, you would find a majority of : them still only run 1.4.3, or maybe 1.9. Even with 2.0, 2.3, or 3.0, that is: still going to be the case. That's probably true, but a nice perk of our current backwards compatibility commitments is that when people pop up asking questions about 1.4.3, we can give them like "upgrading to 2.0.0 solves your problem" and that advice isn't a death sentence -- the steps to move forward are small and easy.I look at things the way things like Maven v1 vs v2 worked out, and how that fractured the community for a long time (as far as i can tell it's still pretty fractured) because the path from v1 to v2 was so steep and involved backtracking so much and i worry that if we make changes to our"copatibility pledge" that don't allow for an even forward walk, we'll wind up with a heavily fractured community. -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
