Questions about doc store files (.cfx)

Hi, I'm wondering about the benefits of having the .cfx files. The main advantage is that you avoid merging (copying) stored fields and TermVectors during segment merge, right? And I think .cfx files are only shared across segments if the same IndexWriter is used to flush multiple segments an

Re: Welcome Robert Muir as Lucene Java core committer!

On 11/8/09 7:05 PM, Grant Ingersoll wrote: - Uwe and his "heavy committing" was worth the 6 hour plane ride each way alone. It never gets old - I can now laugh on cue. More beer is owed ;) Yeah, that and Busch's "UnexpectedSuccessException"... I'm looking forward to that patch, along w/ Ho

Re: Welcome Robert Muir as Lucene Java core committer!

Congratulations Robert! On Fri, Nov 6, 2009 at 5:08 AM, Michael Busch wrote: > I'm excited to announce that the Lucene PMC has voted to promote Robert > Muir to Lucene Java core committer. > Robert was previously contrib committer and it has always been a pleasure > to work with him. > > Please

RE: Build failed in Hudson: Lucene-trunk #1003

No idea, all tests pass here on two machines (Win, Solaris). - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Apache Hudson Server [mailto:hud...@hudson.zones.apache.org] > Sent: Monday, November 09, 2009

Re: Build failed in Hudson: Lucene-trunk #1003

I think it's a bug in the test, plus, the build machine was running so slow that the first doc collected hit the timeout on these threads. It's asserting that the doc that hit the timeout is > 0, and from the test it seems like it intends for the query to never match doc 0, but, the query does in

[jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

[ https://issues.apache.org/jira/browse/LUCENE-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774933#action_12774933 ] Michael McCandless commented on LUCENE-1458: bq. I'll write the flexible index

Re: Questions about doc store files (.cfx)

I think you're asking about the benefit of using "shared doc stores" at all? CFX is just the compound format of these shared files; if compound file is off, then they are still shared, just as separate (.fdx/t, .tvx/d/f) files. For building up a single large index, I suspect the win is sizable, i

[jira] Commented: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774940#action_12774940 ] Simon Willnauer commented on LUCENE-2034: - bq. the testStemExclusionTable( for Bra

[jira] Updated: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2034: Attachment: LUCENE-2034.txt Updated patch to current trunk (the massive patch with @Overri

[jira] Resolved: (LUCENE-2046) IndexReader.isCurrent incorrectly returns false after writer.prepareCommit has been called

[ https://issues.apache.org/jira/browse/LUCENE-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-2046. Resolution: Fixed Thanks Peter! > IndexReader.isCurrent incorrectly returns false

[jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

[ https://issues.apache.org/jira/browse/LUCENE-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774957#action_12774957 ] Mark Miller commented on LUCENE-1458: - bq. Don't you just have to press one button in

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774958#action_12774958 ] Michael McCandless commented on LUCENE-1526: {quote} bq. But, I agree it's was

[jira] Assigned: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler reassigned LUCENE-2041: - Assignee: Uwe Schindler (was: Simon Willnauer) I take it as communicated. > Complete p

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774974#action_12774974 ] Simon Willnauer commented on LUCENE-2041: - thanks uwe! > Complete parallelizaton

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774976#action_12774976 ] Uwe Schindler commented on LUCENE-2041: --- String.isEmpty() is Java 6 only, changed th

[jira] Updated: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-2041: -- Attachment: LUCENE-2041.patch Updated patch. > Complete parallelizaton of ParallelMultiSearch

[jira] Updated: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2041: Attachment: LUCENE-2041.patch Fixed the Java5 issue. Damned I should use java 5 on my not

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774982#action_12774982 ] Simon Willnauer commented on LUCENE-2041: - hehe - my patch was quicker but you com

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774983#action_12774983 ] Uwe Schindler commented on LUCENE-2041: --- I also changed some formatting, we require

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774987#action_12774987 ] Uwe Schindler commented on LUCENE-2041: --- I will also readd support for MaxScore in t

Lucene - Text Classification.

i want to classify sentences stored as strings to a bunch of keywords related to a certain category. so i will have 10 strings which will be a sentence long. and i will want to compare each string to a set of 30 keywords stored somewhere, and then compare with another set of 30 keywords, so on.

Re: Lucene - Text Classification.

Please re-post this question on the lucene user's list, this list is intended for development discussions Best Erick On Mon, Nov 9, 2009 at 10:02 AM, lucenenew wrote: > > i want to classify sentences stored as strings to a bunch of keywords > related > to a certain category. > > so i will h

Re: Questions about doc store files (.cfx)

On 11/9/09 2:56 AM, Michael McCandless wrote: I think you're asking about the benefit of using "shared doc stores" at all? CFX is just the compound format of these shared files; if compound file is off, then they are still shared, just as separate (.fdx/t, .tvx/d/f) files. Oh yeah, that's

[jira] Commented: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775011#action_12775011 ] Robert Muir commented on LUCENE-2034: - simon, good solution. I agree we should depreca

[jira] Commented: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775014#action_12775014 ] Simon Willnauer commented on LUCENE-2034: - bq. i wonder if we should consider a di

Re: Omit positions but not TF

Andrzej Bialecki wrote: Michael McCandless wrote: +1 I guess we'd add a Fieldable.setOmitPositions? And then save that in FieldInfos, and fix the postings writing/reading to respect it? Ie, we can just change the index format. Encoding as negative numbers Yes, that's what I had in mind. I

[jira] Commented: (LUCENE-1458) Further steps towards flexible indexing

[ https://issues.apache.org/jira/browse/LUCENE-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775017#action_12775017 ] Michael McCandless commented on LUCENE-1458: I removed all the "if (Codec.DEBU

[jira] Commented: (LUCENE-1917) ShingleFilter include words

[ https://issues.apache.org/jira/browse/LUCENE-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775019#action_12775019 ] Robert Muir commented on LUCENE-1917: - bq. I'm going to port SOLR-908 rather than reus

[jira] Updated: (LUCENE-2034) Massive Code Duplication in Contrib Analyzers - unifly the analyzer ctors

[ https://issues.apache.org/jira/browse/LUCENE-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2034: Fix Version/s: (was: 3.0) 3.1 Pushed this to 3.1. We might want to

Re: Questions about doc store files (.cfx)

On Mon, Nov 9, 2009 at 10:10 AM, Michael Busch wrote: >> I think you're asking about the benefit of using "shared doc stores" at >> all? >> >> CFX is just the compound format of these shared files; if compound >> file is off, then they are still shared, just as separate (.fdx/t, >> .tvx/d/f) files

Re: Omit positions but not TF

How about opening an issue? This way someone else can come along and pick up the torch... Mike On Mon, Nov 9, 2009 at 11:26 AM, Andrzej Bialecki wrote: > Andrzej Bialecki wrote: >> >> Michael McCandless wrote: >>> >>> +1 >>> >>> I guess we'd add a Fieldable.setOmitPositions?  And then save that

Re: Omit positions but not TF

On Mon, Nov 9, 2009 at 6:03 PM, Michael McCandless wrote: > How about opening an issue?  This way someone else can come along and > pick up the torch... +1 > > Mike > > On Mon, Nov 9, 2009 at 11:26 AM, Andrzej Bialecki wrote: >> Andrzej Bialecki wrote: >>> >>> Michael McCandless wrote:

[jira] Commented: (LUCENE-1260) Norm codec strategy in Similarity

[ https://issues.apache.org/jira/browse/LUCENE-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775046#action_12775046 ] Michael McCandless commented on LUCENE-1260: Has anyone tested performance of

[jira] Assigned: (LUCENE-1260) Norm codec strategy in Similarity

[ https://issues.apache.org/jira/browse/LUCENE-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless reassigned LUCENE-1260: -- Assignee: Michael McCandless > Norm codec strategy in Similarity > ---

[jira] Updated: (LUCENE-1260) Norm codec strategy in Similarity

[ https://issues.apache.org/jira/browse/LUCENE-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-1260: --- Fix Version/s: 3.1 I think this is a reasonable change, but we probably should wait

[jira] Updated: (LUCENE-2022) remove contrib deprecations

[ https://issues.apache.org/jira/browse/LUCENE-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-2022: Attachment: LUCENE-2022.patch patch to remove the rest of the 2.9 deprecations. any 3.0 deprecatio

[jira] Updated: (LUCENE-2022) remove contrib deprecations

[ https://issues.apache.org/jira/browse/LUCENE-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-2022: Attachment: LUCENE-2022.patch also add a "will be removed in 4.0" to the deprecated DistanceApprox

[jira] Commented: (LUCENE-2022) remove contrib deprecations

[ https://issues.apache.org/jira/browse/LUCENE-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775064#action_12775064 ] Ryan McKinley commented on LUCENE-2022: --- For: {panel} also add a "will be removed in

[jira] Updated: (LUCENE-1917) ShingleFilter include words

[ https://issues.apache.org/jira/browse/LUCENE-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated LUCENE-1917: - Fix Version/s: (was: 3.0) 3.1 Moving out of 3.0 > ShingleFilter

[jira] Updated: (LUCENE-1257) Port to Java5

[ https://issues.apache.org/jira/browse/LUCENE-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Kay updated LUCENE-1257: Attachment: LUCENE-1257_o_a_l_demo.patch generics patch for a couple of files > Port to Java5 >

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775076#action_12775076 ] Jason Rutherglen commented on LUCENE-1526: -- {quote} check out the zoie perf pages

[jira] Commented: (LUCENE-2022) remove contrib deprecations

[ https://issues.apache.org/jira/browse/LUCENE-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775077#action_12775077 ] Robert Muir commented on LUCENE-2022: - Ryan, do you think it is best to just remove th

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775082#action_12775082 ] Jake Mannix commented on LUCENE-1526: - I'll try to get those numbers for you, they sho

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775095#action_12775095 ] Michael McCandless commented on LUCENE-1526: {quote} What's missing as it pert

[jira] Created: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

IndexWriter should immediately resolve deleted docs to docID in near-real-time mode --- Key: LUCENE-2047 URL: https://issues.apache.org/jira/browse/LUCENE-2047 Project: L

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775114#action_12775114 ] Michael McCandless commented on LUCENE-1526: OK I opened LUCENE-2047, to resol

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

[ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775115#action_12775115 ] Jason Rutherglen commented on LUCENE-2047: -- Is this going to be an option or defa

[jira] Commented: (LUCENE-2047) IndexWriter should immediately resolve deleted docs to docID in near-real-time mode

[ https://issues.apache.org/jira/browse/LUCENE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775121#action_12775121 ] Michael McCandless commented on LUCENE-2047: I think simply default to true.

[jira] Created: (LUCENE-2048) Omit positions but keep termFreq

Omit positions but keep termFreq Key: LUCENE-2048 URL: https://issues.apache.org/jira/browse/LUCENE-2048 Project: Lucene - Java Issue Type: Improvement Components: Index Affects Versions: 3.1

[jira] Updated: (LUCENE-1313) Near Realtime Search (using a built in RAMDirectory)

[ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rutherglen updated LUCENE-1313: - Attachment: LUCENE-1313.patch This patch includes flushing in a background thread. Some

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775151#action_12775151 ] Jake Mannix commented on LUCENE-1526: - bq. But how many msec does this clone add in pr

[jira] Updated: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-2041: -- Attachment: LUCENE-2041.patch Advanced patch: - Removed @SuppressWarnings by duplicating some

[jira] Updated: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-2041: -- Attachment: LUCENE-2041-refactor.patch Here another possibility, removing more duplicated code

Re: Questions about doc store files (.cfx)

On 11/9/09 9:00 AM, Michael McCandless wrote: Alas, I don't have any benchmarks offhand... if you want to run one, you should be able to hardwire flushDocStores=true in IndexWriter.doFlushInternal? I think that'd turn off the sharing without breaking things (run the tests to be sure ;) ).

[jira] Updated: (LUCENE-1257) Port to Java5

[ https://issues.apache.org/jira/browse/LUCENE-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-1257: Attachment: LUCENE-1257_swing_wikipedia_wordnet_xmlqp.patch contrib/swing,contrib/wikipedia,contri

[jira] Commented: (LUCENE-1812) Static index pruning by in-document term frequency (Carmel pruning)

[ https://issues.apache.org/jira/browse/LUCENE-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775266#action_12775266 ] Robert Muir commented on LUCENE-1812: - Andrzej, i tested your patch. I found two place

Re: Questions about doc store files (.cfx)

On 11/9/09 5:40 PM, Michael Busch wrote: I think that should be ok with parallel indexing, as long as we can always select all corresponding segments from *all* parallel indexes for a merge to keep the docIds in sync. That actually leads me to another question: Let's say you have three segmen

Hudson build is back to normal: Lucene-trunk #1004

See - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org

[jira] Commented: (LUCENE-1260) Norm codec strategy in Similarity

[ https://issues.apache.org/jira/browse/LUCENE-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775310#action_12775310 ] Johan Kindgren commented on LUCENE-1260: Regarding the performance of the TermScor

[jira] Commented: (LUCENE-1526) For near real-time search, use paged copy-on-write BitVector impl

[ https://issues.apache.org/jira/browse/LUCENE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775319#action_12775319 ] John Wang commented on LUCENE-1526: --- bq. I'd love to see how the worst-case queries (mat