[jira] Commented: (LUCENE-2238) deprecate ChineseAnalyzer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806269#action_12806269 ] Simon Willnauer commented on LUCENE-2238: - +1 I will commit this later today if no

[jira] Assigned: (LUCENE-2238) deprecate ChineseAnalyzer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer reassigned LUCENE-2238: --- Assignee: Simon Willnauer > deprecate ChineseAnalyzer > - >

[jira] Resolved: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler resolved LUCENE-2183. --- Resolution: Fixed Committed revision: 904401 Thanks Simon! > Supplementary Character Handli

[jira] Commented: (LUCENE-2238) deprecate ChineseAnalyzer

2010-01-28 Thread Erik Hatcher (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806239#action_12806239 ] Erik Hatcher commented on LUCENE-2238: -- +1 > deprecate ChineseAnalyzer > ---

[jira] Updated: (LUCENE-2238) deprecate ChineseAnalyzer

2010-01-28 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-2238: Attachment: LUCENE-2238.patch > deprecate ChineseAnalyzer > - > >

[jira] Created: (LUCENE-2238) deprecate ChineseAnalyzer

2010-01-28 Thread Robert Muir (JIRA)
deprecate ChineseAnalyzer - Key: LUCENE-2238 URL: https://issues.apache.org/jira/browse/LUCENE-2238 Project: Lucene - Java Issue Type: Task Components: contrib/analyzers Reporter: Robert Muir

[jira] Commented: (LUCENE-2218) ShingleFilter improvements

2010-01-28 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806228#action_12806228 ] Robert Muir commented on LUCENE-2218: - Steven: ok I reviewed this patch, I really like

[jira] Resolved: (LUCENE-2223) ShingleFilter benchmark

2010-01-28 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir resolved LUCENE-2223. - Resolution: Fixed Committed revision 904371. Thanks Steven! > ShingleFilter benchmark > ---

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806143#action_12806143 ] Uwe Schindler commented on LUCENE-2183: --- OK, will commit that tomorrow. Thanks Simon

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Yonik Seeley
On Thu, Jan 28, 2010 at 3:49 PM, Grant Ingersoll wrote: > Could we get the Channel (and other necessary classes) implementation from > Apache Harmony Public JDK methods don't have the low level stuff to implement our own, so rolling our own or using Harmony would require native code... not a goo

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806105#action_12806105 ] Robert Muir commented on LUCENE-2183: - hello simon, thanks for adding this additional

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Yonik Seeley
On Thu, Jan 28, 2010 at 8:24 AM, Michael McCandless wrote: > Bummer. > > So the only viable workarounds are 1) don't use Thread.interrupt (nor, > things like Future.cancel, which in turn use Thread.interrupt) with > NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must > make a de

[jira] Updated: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2183: Attachment: LUCENE-2183.patch Robert / Uwe, I worked on the documentation on method and c

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Grant Ingersoll
Could we get the Channel (and other necessary classes) implementation from Apache Harmony and provide our own implementation? Not ideal, but may solve the problem until it can be resolved by other means. Just a thought, I haven't looked into Harmony's implementation to know whether that is via

[jira] Updated: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-2183: -- Attachment: LUCENE-2183.patch here other javadoc fixes > Supplementary Character Handling in

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806028#action_12806028 ] Simon Willnauer commented on LUCENE-2183: - bq. For that a link using javadoc {...@

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805992#action_12805992 ] Uwe Schindler commented on LUCENE-2183: --- bq. we might want to insert a note/warning

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Robert Muir (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805974#action_12805974 ] Robert Muir commented on LUCENE-2183: - hello, a few very minor nitpicks: * it seems th

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Uwe Schindler (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805935#action_12805935 ] Uwe Schindler commented on LUCENE-2183: --- I'll commit this, if nobody objects in a da

[jira] Commented: (LUCENE-2232) Use VShort to encode positions

2010-01-28 Thread Paul Elschot (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805916#action_12805916 ] Paul Elschot commented on LUCENE-2232: -- Meanwhile I've been looking around some more

[jira] Updated: (LUCENE-2232) Use VShort to encode positions

2010-01-28 Thread Paul Elschot (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Elschot updated LUCENE-2232: - Attachment: LUCENE-2232-nonbackwards.patch Clean patch changing position delta encoding from vby

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Michael McCandless
Bummer. So the only viable workarounds are 1) don't use Thread.interrupt (nor, things like Future.cancel, which in turn use Thread.interrupt) with NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the app must make a deletion policy that keeps a commit alive if any reader is using it. Or,

[jira] Issue Comment Edited: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805909#action_12805909 ] Simon Willnauer edited comment on LUCENE-2183 at 1/28/10 1:16 PM: --

[jira] Commented: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805909#action_12805909 ] Simon Willnauer commented on LUCENE-2183: - I did run following benchmark alg file

[jira] Updated: (LUCENE-2183) Supplementary Character Handling in CharTokenizer

2010-01-28 Thread Simon Willnauer (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Willnauer updated LUCENE-2183: Attachment: LUCENE-2183.patch Added CHANGES.TXT entry and fixed 2 supplementary chars rela

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Simon Willnauer
On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless wrote: > On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler wrote: > >> So I checked the code of NIOFSIndexInput, my last comment was not really >> correct: >> NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the >> ctor RAF.

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Michael McCandless
On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler wrote: > So I checked the code of NIOFSIndexInput, my last comment was not really > correct: > NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the > ctor RAF.getChannel() is called. The RAF keeps open until the file is closed

[jira] Resolved: (LUCENE-2217) Remaining reallocation should use ArrayUtil.getNextSize()

2010-01-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-2217. Resolution: Fixed Fix Version/s: 3.1 Thanks Paul! > Remaining reallocation

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Michael McCandless
Yeah, so not great (doubling your file descriptor usage to protect against this). Still we could provide that as an option to NIOFSDir... We could at least attempt the reopen, in case the file is present. This way, at least, one could make a (custom, app dependent) deletion policy that ensure any

RE: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Uwe Schindler
Hi again, So I checked the code of NIOFSIndexInput, my last comment was not really correct: NIOFSIndexInput extends SimpleFSIndexInput and that opens the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open until the file is closed (and also the channel). So it's really simple to fi

RE: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Uwe Schindler
>From JavaDocs: The state of a file channel is intimately connected to that of the object whose getChannel method returned the channel. Changing the channel's position, whether explicitly or by reading or writing bytes, will change the file position of the originating object, and vice versa. Cha

RE: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Uwe Schindler
I am not sure, because RandomAccessFile has a getChannel method, so I suspect that they share the descriptor. Maybe check it with lsof? Possibly we could wait until Simon provides a testcase that fails. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@t

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Simon Willnauer
for sure it will consume 2 file descriptors and it requires some workaround that checks if we are hitting this exception. kind of ugly anyway. On Thu, Jan 28, 2010 at 12:07 PM, Michael McCandless wrote: > On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler wrote: >> But if you keep the underlying Ra

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Michael McCandless
On Thu, Jan 28, 2010 at 5:59 AM, Uwe Schindler wrote: > But if you keep the underlying RandomAccessFile open? We could do that but... won't this consume 2 file descriptors? Mike - To unsubscribe, e-mail: java-dev-unsubscr...@lu

RE: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Uwe Schindler
But if you keep the underlying RandomAccessFile open? - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Michael McCandless [mailto:luc...@mikemccandless.com] > Sent: Thursday, January 28, 2010 11:47 AM > To

[jira] Updated: (LUCENE-2111) Wrapup flexible indexing

2010-01-28 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-2111: --- Attachment: LUCENE-2111.patch Attached patch with a major reworking of some parts of

Re: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Michael McCandless
On Thu, Jan 28, 2010 at 5:20 AM, Uwe Schindler wrote: > Can we fix NIOFSIndexInput to simply reopen the channel when the exception > occurs? The problem is that the file may have been deleted in the meantime. This is quite a nasty behavior of FileChannel. Mike ---

RE: Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Uwe Schindler
Can we fix NIOFSIndexInput to simply reopen the channel when the exception occurs? - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Simon Willnauer [mailto:simon.willna...@googlemail.com] > Sent: Thursday

Nasty NIO behavior makes NIOFSDirectory silently close channel

2010-01-28 Thread Simon Willnauer
Last week I ran into nasty hardly reproducible ChannelClosedExceptions thrown by NIOFSDirectory#readInternal() during term position reading. I had a hard time to figure out what causes the channel to be closed as the IndexReader where I obtained the Indexinput or rather TermPositions from was still