[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850726#action_12850726
]
Earwin Burrfoot commented on LUCENE-2345:
-
Just created LUCENE-2355 to track reopen stuff. No issue for plugin model yet,
but I'll probably create it, can't edit this one, I'm no committer.
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch, LUCENE-2345_3.0.plugins.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850062#action_12850062
]
Michael McCandless commented on LUCENE-2345:
I think we should only commit this only on 3.1 (new feature)?
Earwin: will this change really conflict w/ your ongoing refactoring (to have
DirReader subclass MultiReader)? It seems somewhat orthogonal?
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850067#action_12850067
]
Earwin Burrfoot commented on LUCENE-2345:
-
bq. will this change really conflict w/ your ongoing refactoring (to have
DirReader subclass MultiReader)? It seems somewhat orthogonal?
Ongoing contains a massive rework of how open/reopen/clone/close is done.
Folding copypaste between DirReader and MultiReader is a byproduct.
The aim is to have clean initialization, get rid of of init/reinit/moreinit
methods, moving the code to constructors.
This alone plays bad with factories.
The next step, plugins - conflicts ideologically. Plugins allow extension by
composition, which is (in my view) much more clean than subclassing.
Landing them without refactoring all this mess beforehand is dangerous, I
already hit some places where reopen/clone were done wrong in respect to
existing reader fields. Even more - NRT reader lifecycle is really messy - it
beginnings as a reader aimed only for merging, and then being upgraded in two
separate stages is finally wrapped in DirReader to be used for actual
searching. The factory/init() approach ignores this, and each user of this API
will be on his own to separate lightweight readers from full-fledged ones.
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850075#action_12850075
]
Shai Erera commented on LUCENE-2345:
Earwin, w/o knowing too much about the details of your work, I wanted to
comment on get rid of of init/reinit/moreinit methods, moving the code to
constructors. I work now on Parallel Index and one of the things I do is
extend IW. Currently, IW's ctor code performs the initialization, however I'm
thinking to move that code to an init method. The reason is to allow easy
extensions of IW, such as LUCENE-2330. There I'm going to add a default ctor to
IW, accompanied by an init method the extending class can call if needed. So
what I'm trying to say is that init methods are not always bad, and sometimes
ctors limit you. Perhaps it would make sense though in what you're trying to do
...
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850079#action_12850079
]
Uwe Schindler commented on LUCENE-2345:
---
Non-final/private method calls from ctors is discouraged, as it creates
problems:
- If you override such a init method, you can change the behaviour of the super
class' ctor
- During the time the init() method is called, the fields are not yet
initialized in the subclass and also the subclass' ctor was not yet called!
This leads to unexspected behaviour (like final fields changing its contents
during the lifetime of the class: during the overridden init() method they are
null and suddenly change to non-null after the subclass' ctor was executed,
which is after the init() method)
Because of this, such init() methods should be private or at least final and
never public!
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850081#action_12850081
]
Uwe Schindler commented on LUCENE-2345:
---
Here a good explanation: [http://www.informit.com/articles/article.aspx?p=20521]
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850082#action_12850082
]
Michael McCandless commented on LUCENE-2345:
OK, I guess for now let's hold off on this issue while we iterate at least with
Earwin's first refactoring effort...?
Earwin, can you post a patch w/ your current state, even if it's rough?
There seems to be alot of interest/opinions on how to fix things here :)
Both IndexWriter SegmentReader are needing extensibility... but this is a
big change so I'm hoping we can first just get your refactoring done before
adding extensibility.
Tim, do you think the plugin model (extension by composition) would be
workable for your use case? Ie, instead of a factory enabling subclasses of
SegmentReader?
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850083#action_12850083
]
Shai Erera commented on LUCENE-2345:
Thanks Uwe, I know that ctor is the preferred way, and in the process of
introducing IWC I delete IW.init which all ctors called and pulled all the code
to IW ctor. I will make that init() on IW final. But sometimes putting code in
init() is not bad (and it's used in Lucene elsewhere too (e.g. PQ and up until
recently IW).
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850084#action_12850084
]
Earwin Burrfoot commented on LUCENE-2345:
-
More often than not init() methods are a sign of bad design.
I.e. in your case extending IW is crazy.
You should have an interface capturing IW methods and two implementations - one
writing to the index and another delegating to its subwriters. You don't do
DirectoryReader extends SegmentReader, do you? They both extend
lucene-style-interface IndexReader.
Lucene's back compat policy got people used to writing and digesting freaky
code, and I'm not going to fight generally against it, that's a lost cause : )
But in this exact case (readers) stuffing everything in constructors, defining
all fields I can final, replacing SR.openDocStores/loadTermsIndex with
reopen()-like method allows me to tackle (at least somewhat) lifecycle
complexity. Javac will force me to either initialize something or explicitly
leave it null.
Besides some real bugs in this code, I cleaned up cases where a field was
inited twice, just in case! Poor developers got lost in init() methods : }
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850088#action_12850088
]
Earwin Burrfoot commented on LUCENE-2345:
-
bq. Earwin, can you post a patch w/ your current state, even if it's rough?
Ahem. Right now it's more or less finished for
Multi/Directory/MutableDirectory/WriterBackedDirectory-readers.
But the SegmentReader is in shambles (i.e. does not compile yet).
Should I post asis?
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850094#action_12850094
]
Shai Erera commented on LUCENE-2345:
Earwin, I wholeheartedly agree with what you wrote. If we could refactor IW and
extract it to a set of interfaces, then I agree (and Michael B. has an issue
open for that). I think though that IW's API is already that interface (give or
take few methods). So perhaps this can be an easy refactoring - introduce an
Indexer (a la Searcher) class (or interface) w/ all of IW public methods, and
then let PW extend/impl that class/interface as well as IW. We can also
consider making IW itself final this way (though bw police will prevent it :)).
Then when PW sets up the slices, it can create them as IW or any other IW-like
implementation it needs them to impl. If it sounds good enough to become its
own issue, I can open one and we can continue discussing it there (and leave
that issue focused on extending SR). Then I'll hold off w/ LUCENE-2330, or
simply rename it to reflect that Indexer API.
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850127#action_12850127
]
Tim Smith commented on LUCENE-2345:
---
bq. I think we should only commit this only on 3.1 (new feature)?
3.1 only of course (just posted a 3.0 patch now as that's what i'm using and i
need the functionality now)
bq. Tim, do you think the plugin model (extension by composition) would be
workable for your use case? Ie, instead of a factory enabling subclasses of
SegmentReader?
As long as the plugin model allows the same capabilities, that could work just
fine and could be the final solution for this ticket.
I mainly need the ability to add data structures to a SegmentReader that will
be shared for all SegmentReader's for a segment, and then add some extra meta
information on a per instance basis
Is there a ticket or wiki page that details the plugin architecture/design so
i could take a look?
However, would the plugins allow overriding specific IndexReader methods?
I still would see the need to be able to override specific methods for a
SegmentReader (in order to track statistics/provide
changed/different/faster/more feature rich implementations)
I don't have a direct need for this right now, however i could envision needing
this in the future
Here's a few requirements i would pose for the plugin model (maybe they are
already though of):
* Plugins have hooks to reopen themselves (some plugins can be shared across
all instances of a SegmentReader)
** These reopen hooks would be called during SegmentReader.reopen()
* Plugins are initialized during SegmentReader.get/SegmentReader.reopen
** plugins should not have to be added after the fact, as this would not allow
proper warming/initializing of plugins inside the NRT indexing
** i assume this would need be added as some list of PluginFactories added to
the IndexWriter/IndexReader.open()?
* Plugins should have a close method that is called in SegmentReader.close()
** This will allow proper release of any resources
* Plugins are passed an instance of the SegmentReader they are for
** Plugins should be able to access all methods on a SegmentReader
** This would effectively allow overriding a SegmentReader by having a plugin
provide the functionality instead (however only people explicitly calling the
plugin would get this benefit)
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850323#action_12850323
]
Tim Smith commented on LUCENE-2345:
---
found one issue with the plugins patch
With NRT indexing, if the SegmentReader is opened with no TermInfosReader (for
merging), then the plugins will be initialized with a SegmentReader that has no
ability to walk the TermsEnum.
I guess SegmentPlugin initialization should wait until after the terms index is
loaded or have another method for catching this event to the SegmentPlugin
interface
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch, LUCENE-2345_3.0.plugins.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850348#action_12850348
]
Earwin Burrfoot commented on LUCENE-2345:
-
I mentioned this issue earlier. My patch removes loadTermsIndex method from
SegmentReader and requires you to reopen it. At that moment you can stuff it
with a set of plugins without leaving 'final' paradise.
There are still things to consider. Some SR guts could be converted to plugins
themselves, so you can override them with your implementation if you wish. If
that is so, there should be a way to decide which of the plugins should be
loaded for current SR mode. My previous design loaded plugins only for
full-fledged readers.
bq. Is there a ticket or wiki page that details the plugin
architecture/design so i could take a look?
The design itself was never published, rather it was discussed several times on
a mailing list, and simmered inside my head for some time. I have a first impl
running at my workplace, but it is really fugly :) I will flesh out a proper
description after the refactoring, but it has all your points in one or another
form. Biggest differences are - they are keyed by Class not by String, they
declare multiple interfaces they provide, they declare dependencies, they don't
have to implement/extend anything, as all the hooks are on the factory classes.
Also, they extend not only SegmentReader, but the whole hierarchy - SR, MR, DR,
whatever.
The problem of overriding SR methods is solved by delegating these methods to
plugins, which can be either default or user-provided. (But remember the
question of which subset should be initialized for which SR mode)
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch, LUCENE-2345_3.0.plugins.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850356#action_12850356
]
Marvin Humphrey commented on LUCENE-2345:
-
Is there a ticket or wiki page that details the plugin architecture/design
so i could take a look?
FWIW, KinoSearch has a complete prototype implementation of this design,
based loosely on the mailing list conversations that Earwin referred to.
* SegReader and SegWriter are both composites with minimal APIs.
* All subcomponents subclass either DataWriter or DataReader.
* The Architecture class (under KinoSearch::Plan) determines which plugins
get loaded.
[http://www.rectangular.com/svn/kinosearch/trunk/core/]
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch, LUCENE-2345_3.0.plugins.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12850361#action_12850361
]
Tim Smith commented on LUCENE-2345:
---
bq. My patch removes loadTermsIndex method from SegmentReader and requires you
to reopen it.
that's definitely much cleaner and would solve the issue in my current patch
(sadly i'm on 3.0 and want to keep my patch there at a minimum until i can port
to all the goodness on 3.1).
bq. Also, they extend not only SegmentReader, but the whole hierarchy - SR, MR,
DR, whatever.
i just wussed out and just did only the SegmentReader case as thats all i need
right now
bq. as all the hooks are on the factory classes
could you post your factory class interface?
If i base my 3.0 patch off that i can reduce my 3.1 port overhead.
are there any tickets tracking your reopen refactors or your plugin model?
If not, feel free to retool this ticket for your plugin model for Index Readers
as that will solve my use cases (and then some)
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch, LUCENE-2345_3.0.plugins.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849728#action_12849728
]
Shai Erera commented on LUCENE-2345:
bq. The IndexWriter now has a getter and setter for setting this
If this is not expected to change during the lifetime of IW, I think it should
be added to IWC when you upgrade the patch to 3.1.
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849731#action_12849731
]
Tim Smith commented on LUCENE-2345:
---
that was my plan
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
Attachments: LUCENE-2345_3.0.patch
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849450#action_12849450
]
Michael McCandless commented on LUCENE-2345:
+1
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849455#action_12849455
]
Tim Smith commented on LUCENE-2345:
---
that's the reassurance i needed :)
will start working on a patch tomorrow
will take a few days as i'll start with a 3.0 patch (which i use), then will
create a 3.1 patch once i've got that all flushed out
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849475#action_12849475
]
Earwin Burrfoot commented on LUCENE-2345:
-
Ahem, that directly clashes with my inflight patch?
I'm going to land create/reopen/close refactoring pretty soon, and then finish
plugins.
(Which was an arduous task before, as initialization paths for SegmentReaders
are really, really insane)
I'm on this patch for quite some time, I understand that, so maybe we can
settle on a deadline? Don't like to see the effort vaporate, also merging is
gonna be hell with flex branch alone, don't want to double it :)
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2345) Make it possible to subclass SegmentReader
[
https://issues.apache.org/jira/browse/LUCENE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849497#action_12849497
]
Tim Smith commented on LUCENE-2345:
---
i'll do my initial work on 3.0 so i can absorb the changes now and will post
that patch
at which point, i can wait for you to finish whatever you need, or we can just
incorporate the same ability into your patch for the other ticket
i would just like to see the ability to subclass SegmentReader's on 3.1 so i
don't have to port a patch when i absorb 3.1 (just use the finalized apis)
Make it possible to subclass SegmentReader
--
Key: LUCENE-2345
URL: https://issues.apache.org/jira/browse/LUCENE-2345
Project: Lucene - Java
Issue Type: Wish
Components: Index
Reporter: Tim Smith
Fix For: 3.1
I would like the ability to subclass SegmentReader for numerous reasons:
* to capture initialization/close events
* attach custom objects to an instance of a segment reader (caches,
statistics, so on and so forth)
* override methods on segment reader as needed
currently this isn't really possible
I propose adding a SegmentReaderFactory that would allow creating custom
subclasses of SegmentReader
default implementation would be something like:
{code}
public class SegmentReaderFactory {
public SegmentReader get(boolean readOnly) {
return readOnly ? new ReadOnlySegmentReader() : new SegmentReader();
}
public SegmentReader reopen(SegmentReader reader, boolean readOnly) {
return newSegmentReader(readOnly);
}
}
{code}
It would then be made possible to pass a SegmentReaderFactory to IndexWriter
(for pooled readers) as well as to SegmentReader.get() (DirectoryReader.open,
etc)
I could prepare a patch if others think this has merit
Obviously, this API would be experimental/advanced/will change in future
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
