Yup - you need for anything developed outside of Apache. Michael McCandless wrote: > Well, it's the usual process... pull together a big patch, open an issue, etc. > > Probably because it's a large amount of code (I think?) you'll need to > submit a software grant > (http://www.apache.org/licenses/software-grant.txt). > > Mike > > On Thu, Oct 8, 2009 at 2:58 PM, John Wang <john.w...@gmail.com> wrote: > >> Awesome! >> >> Mike, can you let us know what the process is and the time line? >> >> Thanks >> >> -John >> >> On Thu, Oct 8, 2009 at 11:48 AM, Michael McCandless >> <luc...@mikemccandless.com> wrote: >> >>> +1! >>> >>> Mike >>> >>> On Thu, Oct 8, 2009 at 2:41 PM, John Wang <john.w...@gmail.com> wrote: >>> >>>> Hi guys: >>>> >>>> What are your thoughts about contributing Kamikaze as a lucene >>>> contrib >>>> package? We just finished porting kamikaze to lucene 2.9. With the new >>>> 2.9 >>>> api, it allows us for some more code tuning and optimization >>>> improvements. >>>> >>>> We will be releasing kamikaze, it might a good time to add it to >>>> the >>>> lucene contrib package if there is interest. >>>> >>>> Thanks >>>> >>>> -John >>>> >>>> On Thu, Sep 24, 2009 at 6:20 AM, Uwe Schindler <u...@thetaphi.de> wrote: >>>> >>>>> By the way: In the last RC of Lucene 2.9 we added a new method to >>>>> DocIdSet >>>>> called isCacheable(). It is used by e.g. CachingWrapperFilter to >>>>> determine, >>>>> if a DocIdSet is easy cacheable or must be copied to an OpenBitSetDISI >>>>> (the >>>>> default is false, so all custom DocIdSets are copied to OpenBitSetDISI >>>>> by >>>>> CachingWrapperFilter, even if not needed - if a DocIdSet does not do >>>>> disk >>>>> IO >>>>> and have a fast iterator like e.g. the FieldCache ones in >>>>> FieldCacheRangeFilter, it should return true; see CHANGES.txt). Maybe >>>>> this >>>>> should also be added to Kamikaze, which is a really nice project! >>>>> Especially >>>>> filter DocIdSets should pass this method to its delegate (see >>>>> FilterDocIdSet >>>>> in Lucene). >>>>> >>>>> ----- >>>>> Uwe Schindler >>>>> H.-H.-Meier-Allee 63, D-28213 Bremen >>>>> http://www.thetaphi.de >>>>> eMail: u...@thetaphi.de >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: John Wang (JIRA) [mailto:j...@apache.org] >>>>>> Sent: Thursday, September 24, 2009 3:14 PM >>>>>> To: java-dev@lucene.apache.org >>>>>> Subject: [jira] Commented: (LUCENE-1458) Further steps towards >>>>>> flexible >>>>>> indexing >>>>>> >>>>>> >>>>>> [ https://issues.apache.org/jira/browse/LUCENE- >>>>>> 1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment- >>>>>> tabpanel&focusedCommentId=12759112#action_12759112 ] >>>>>> >>>>>> John Wang commented on LUCENE-1458: >>>>>> ----------------------------------- >>>>>> >>>>>> Just a FYI: Kamikaze was originally started as our sandbox for Lucene >>>>>> contributions until 2.4 is ready. (we needed the DocIdSet/Iterator >>>>>> abstraction that was migrated from Solr) >>>>>> >>>>>> It has three components: >>>>>> >>>>>> 1) P4Delta >>>>>> 2) Logical boolean operations on DocIdSet/Iterators (I have created a >>>>>> jira >>>>>> ticket and a patch for Lucene awhile ago with performance numbers. It >>>>>> is >>>>>> significantly faster than DisjunctionScorer) >>>>>> 3) algorithm to determine which DocIdSet implementations to use given >>>>>> some >>>>>> parameters, e.g. miniD,maxid,id count etc. It learns and adjust from >>>>>> the >>>>>> application behavior if not all parameters are given. >>>>>> >>>>>> So please feel free to incorporate anything you see if or move it to >>>>>> contrib. >>>>>> >>>>>> >>>>>> >>>>>>> Further steps towards flexible indexing >>>>>>> --------------------------------------- >>>>>>> >>>>>>> Key: LUCENE-1458 >>>>>>> URL: >>>>>>> https://issues.apache.org/jira/browse/LUCENE-1458 >>>>>>> Project: Lucene - Java >>>>>>> Issue Type: New Feature >>>>>>> Components: Index >>>>>>> Affects Versions: 2.9 >>>>>>> Reporter: Michael McCandless >>>>>>> Assignee: Michael McCandless >>>>>>> Priority: Minor >>>>>>> Attachments: LUCENE-1458-back-compat.patch, >>>>>>> LUCENE-1458-back- >>>>>>> >>>>>> compat.patch, LUCENE-1458-back-compat.patch, LUCENE-1458.patch, >>>>>> LUCENE- >>>>>> 1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, LUCENE-1458.patch, >>>>>> LUCENE-1458.patch, LUCENE-1458.tar.bz2, LUCENE-1458.tar.bz2, LUCENE- >>>>>> 1458.tar.bz2, LUCENE-1458.tar.bz2 >>>>>> >>>>>>> I attached a very rough checkpoint of my current patch, to get >>>>>>> early >>>>>>> feedback. All tests pass, though back compat tests don't pass due >>>>>>> to >>>>>>> changes to package-private APIs plus certain bugs in tests that >>>>>>> happened to work (eg call TermPostions.nextPosition() too many >>>>>>> times, >>>>>>> which the new API asserts against). >>>>>>> [Aside: I think, when we commit changes to package-private APIs >>>>>>> such >>>>>>> that back-compat tests don't pass, we could go back, make a branch >>>>>>> on >>>>>>> the back-compat tag, commit changes to the tests to use the new >>>>>>> package private APIs on that branch, then fix nightly build to use >>>>>>> the >>>>>>> tip of that branch?o] >>>>>>> There's still plenty to do before this is committable! This is a >>>>>>> rather large change: >>>>>>> * Switches to a new more efficient terms dict format. This still >>>>>>> uses tii/tis files, but the tii only stores term & long offset >>>>>>> (not a TermInfo). At seek points, tis encodes term & freq/prox >>>>>>> offsets absolutely instead of with deltas delta. Also, tis/tii >>>>>>> are structured by field, so we don't have to record field >>>>>>> number >>>>>>> in every term. >>>>>>> . >>>>>>> On first 1 M docs of Wikipedia, tii file is 36% smaller (0.99 >>>>>>> MB >>>>>>> -> 0.64 MB) and tis file is 9% smaller (75.5 MB -> 68.5 MB). >>>>>>> . >>>>>>> RAM usage when loading terms dict index is significantly less >>>>>>> since we only load an array of offsets and an array of String >>>>>>> (no >>>>>>> more TermInfo array). It should be faster to init too. >>>>>>> . >>>>>>> This part is basically done. >>>>>>> * Introduces modular reader codec that strongly decouples terms >>>>>>> dict >>>>>>> from docs/positions readers. EG there is no more TermInfo used >>>>>>> when reading the new format. >>>>>>> . >>>>>>> There's nice symmetry now between reading & writing in the >>>>>>> codec >>>>>>> chain -- the current docs/prox format is captured in: >>>>>>> {code} >>>>>>> FormatPostingsTermsDictWriter/Reader >>>>>>> FormatPostingsDocsWriter/Reader (.frq file) and >>>>>>> FormatPostingsPositionsWriter/Reader (.prx file). >>>>>>> {code} >>>>>>> This part is basically done. >>>>>>> * Introduces a new "flex" API for iterating through the fields, >>>>>>> terms, docs and positions: >>>>>>> {code} >>>>>>> FieldProducer -> TermsEnum -> DocsEnum -> PostingsEnum >>>>>>> {code} >>>>>>> This replaces TermEnum/Docs/Positions. SegmentReader emulates >>>>>>> the >>>>>>> old API on top of the new API to keep back-compat. >>>>>>> >>>>>>> Next steps: >>>>>>> * Plug in new codecs (pulsing, pfor) to exercise the modularity / >>>>>>> fix any hidden assumptions. >>>>>>> * Expose new API out of IndexReader, deprecate old API but >>>>>>> emulate >>>>>>> old API on top of new one, switch all core/contrib users to the >>>>>>> new API. >>>>>>> * Maybe switch to AttributeSources as the base class for >>>>>>> TermsEnum, >>>>>>> DocsEnum, PostingsEnum -- this would give readers API >>>>>>> flexibility >>>>>>> (not just index-file-format flexibility). EG if someone wanted >>>>>>> to store payload at the term-doc level instead of >>>>>>> term-doc-position level, you could just add a new attribute. >>>>>>> * Test performance & iterate. >>>>>>> >>>>>> -- >>>>>> This message is automatically generated by JIRA. >>>>>> - >>>>>> You can reply to this email to add a comment to the issue online. >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>>>>> For additional commands, e-mail: java-dev-h...@lucene.apache.org >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>>>> For additional commands, e-mail: java-dev-h...@lucene.apache.org >>>>> >>>>> >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-dev-h...@lucene.apache.org >>> >>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >
-- - Mark http://www.lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
