[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560036#comment-15560036
]
Yonik Seeley commented on LUCENE-7283:
--
bq. I think it's really time to get rid of FieldCache
I may have missed it, but I didn't see any mailing list discussion (or any
other discussion) around this.
The main takeaway here is a reminder that consensus around bigger decisions
needs to be done in the open.
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15559753#comment-15559753
]
Adrien Grand commented on LUCENE-7283:
--
On the other hand, doc values have been there for 4 years now. Like Solr,
Elasticsearch has some history with FieldCache which makes the removal take
time. But it is on the way out: the upcoming release based on Lucene 6 will
only allow it on text fields, and as an opt-in (disabled by default) and all
other fields (numerics, keywords, etc.) will enforce usage of doc values for
sorting or faceting. And I truly hope that we can entirely get rid of it when
we upgrade to Lucene 7. Now that we have had doc values for 4 years, I think
it's really time to get rid of FieldCache. Apps that truly need it for a bit
longer for backward compatibility guarantees can still fork the code like Mike
suggested.
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558656#comment-15558656
]
Yonik Seeley commented on LUCENE-7283:
--
bq. In ancient Lucene versions, sorting was not a "first class citizen" and
hacks like field cache were necessary.
Yes, a hack that has been with us for more than *12 years* already (FieldCache
came about in 2004).
Best practice is one thing (Solr uses docvalues by default), but removing the
option is different.
After a quick google, it doesn't appear like you're removing this ability from
elasticsearch (fielddata is the ES equivalent of the Lucene FieldCache):
https://www.elastic.co/guide/en/elasticsearch/reference/5.x/modules-fielddata.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558552#comment-15558552
]
Michael McCandless commented on LUCENE-7283:
bq. I imagine they will be quite surprised to see everything about the
FieldCache removed and left with no ability to Sort on an indexed-only field.
I doubt that.
In ancient Lucene versions, sorting was not a "first class citizen" and hacks
like field cache were necessary.
But then, long ago, we added doc values, and we've been saying for years now
that if you want to sort by X you need to index X as doc values. This change
(in 7.0 only, just deprecated in 6.x) should come as a surprise to nobody.
Doc values are far more efficient than the the legacy field cache: no up front
penalty the first time you sort on the field, no massive heap usage, no heavy
full GC cost. Instead the OS manages the hot pages and swaps as necessary,
thanks to efficient doc values codec formats.
And such theoretical users have plenty of options: they can use Solr, or only
use {{UninvertingReader}} or {{SlowCompositeReaderWrapper}} out of Solr's
sources, or fork them for their own purposes, or reindex with the
much-more-efficient doc values.
Such horribly inefficient legacy code really does not belong in Lucene (or
Solr!).
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558249#comment-15558249
]
Yonik Seeley commented on LUCENE-7283:
--
bq. p.s. please update the description to include UninvertingReader (which is
much larger in scope than just SCW and unrelated to SCW)
Indeed, the original JIRA description of "Move SlowCompositeReaderWrapper to
solr sources" doesn't begin to describe the impact this issue will have on
normal Lucene users in 7.0. I imagine they will be quite surprised to see
everything about the FieldCache removed and left with no ability to Sort on an
indexed-only field.
It really seems like there should be a lot more consensus around removing
functionality on this scale.
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7283) Move SlowCompositeReaderWrapper and uninverting package to solr sources
[
https://issues.apache.org/jira/browse/LUCENE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304307#comment-15304307
]
ASF subversion and git services commented on LUCENE-7283:
-
Commit 5525f429288cf8480ae7b6dc1438918e809a242c in lucene-solr's branch
refs/heads/branch_6x from [~yo...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5525f42 ]
SOLR-9160: Sync 6x and 7.0 move of UninvertingReader,
SlowCompositeReaderWrapper for Solr (LUCENE-7283)
> Move SlowCompositeReaderWrapper and uninverting package to solr sources
> ---
>
> Key: LUCENE-7283
> URL: https://issues.apache.org/jira/browse/LUCENE-7283
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: 6.1, master (7.0)
>
> Attachments: LUCENE-7283.patch
>
>
> Spinoff from LUCENE-6766, where we fixed index-time sorting to have first
> class support in Lucene's ore, and no longer use
> {{SlowCompositeReaderWrapper}}.
> This is a dangerous, long living class, that tries to pretend a set of N
> segments is actually just a single segment. It's a leaky abstraction, has
> poor performance, and puts undue pressure on the APIs of new Lucene features
> to try to keep up this illusion.
> With LUCENE-6766, finally all usage of this class (except for
> {{UninvertedReader}} tests, which should maybe also move out?) has been
> removed from Lucene, so I think we should move it to Solr. This may also
> lead to a solution for LUCENE-7086 since e.g. the class could tap into solr's
> schema to "know" how to handle points fields properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
