RE: Breaking Java back-compat in Solr

[Martin Gainty]

From: ser...@gmail.com
Date: Sun, 10 Jan 2016 13:58:52 +
Subject: Re: Breaking Java back-compat in Solr
To: dev@lucene.apache.org

I think that our Maven integration only publishes Lucene/Solr artifacts to 
Maven central, but I don't think the POMs actually produce the artifacts? I 
could be wrong though ...MG>at least *not yet*

If that's the case, then we'd need to only add these to Solr's build.xml, like 
it's done in Lucene.
MG>common-build.xml jarify macrodef seems to do the heavy lifting to aggregate 
the jar  
 
   
   
 

  

MG>thanks shai...Martin
Shai

On Sun, Jan 10, 2016 at 3:35 PM Martin Gainty <mgai...@hotmail.com> wrote:



can you also add this to lucene pom.xml ?

Martin 
__ 

  


From: ser...@gmail.com
Date: Sun, 10 Jan 2016 07:57:26 +0000
Subject: Re: Breaking Java back-compat in Solr
To: dev@lucene.apache.org

As for SOLR-8475, I will commit it to trunk only. But I think that we should 
come up w/ concrete annotations in our code, and annotate classes as we go, to 
back our back-compat policy. I propose that we add these:

@solr.internal - this is internal API and will change without notice. No 
back-compat guarantees.
@solr.experimental - this is a candidate for a new public API, but will likely 
change until it stabilizes. No (strong) back-compat guarantees.
@solr.expert - while you can use this API, we cannot guarantee strong 
back-compat support. Will be on a per-case basis.
@solr.public - this is our public API, standard back-compat policy. Of course, 
if something needs break, we'll discuss the case, but otherwise users who use 
this API can expect to upgrade minor releases without re-compiling their code. 
Immediate candidate is SolrJ.

I also propose that until we tag all classes, we treat "no annotation" as 
@solr.internal (except for SolrJ code). That will force us to explicitly tag 
classes that we think should at least be @solr.public, since that's the only 
annotation that should draw the attention of developers.

If people agree, we can add these annotations to the build.xml, to add proper 
text to javadocs.

Shai

On Sat, Jan 9, 2016 at 2:15 AM Jack Krupansky <jack.krupan...@gmail.com> wrote:
With the talk of 6.0 coming out real soon and not waiting for new work, will 
this 6.0/5.x issue become moot and morph into an issue for 7.0/6.x?
Settling the criteria for Solr plugin API back-compat still seems urgent, but 
if the SOLR-8475 work can quickly get committed to trunk for 6.0 maybe that 
takes some of the pressure off. Still, I'd prefer that the back-compat criteria 
be settled ASAP.
-- Jack Krupansky

On Wed, Jan 6, 2016 at 10:43 AM, Yonik Seeley <ysee...@gmail.com> wrote:
On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta <ans...@anshumgupta.net> wrote:

> As I understand, seems like there's reasonable consensus that we will:

>

> 1. provide strong back-compat for for SolrJ and REST APIs

> 2. Strive to maintain but not guarantee *strong* back-compat for Java APIs.



I think this actually represents what our current policy already is.

The sticking point is perhaps "Strive to maintain" is changing

definition to become much more lenient, to the point of being

meaningless.



Let's look at the issue that spawned this thread:

https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to

SolrIndexSearcher)



The issue is if QueryCommand and QueryResult should be moved out of

SolrIndexSearcher in 5.x (essentially a rename), or of that rename

should only be in 6.0.  If one's desire for a class rename (of classes

that are likely to be used by plugins) overrides #2, I'd argue that

means we essentially have no #2 at all.  Or perhaps I'm not grasping

why it's really that important to rename those classes.



Regarding annotations:

Multiple people have suggested annotating classes that should remain

back compat.  If we were to do this, wouldn't we want those

annotations to cover the classes in question

(SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they

cover and still be useful?



-Yonik



-

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org

For additional commands, e-mail: dev-h...@lucene.apache.org




  

RE: Breaking Java back-compat in Solr

[Martin Gainty]

can you also add this to lucene pom.xml ?
Martin 
__ 

  


From: ser...@gmail.com
Date: Sun, 10 Jan 2016 07:57:26 +
Subject: Re: Breaking Java back-compat in Solr
To: dev@lucene.apache.org

As for SOLR-8475, I will commit it to trunk only. But I think that we should 
come up w/ concrete annotations in our code, and annotate classes as we go, to 
back our back-compat policy. I propose that we add these:

@solr.internal - this is internal API and will change without notice. No 
back-compat guarantees.
@solr.experimental - this is a candidate for a new public API, but will likely 
change until it stabilizes. No (strong) back-compat guarantees.
@solr.expert - while you can use this API, we cannot guarantee strong 
back-compat support. Will be on a per-case basis.
@solr.public - this is our public API, standard back-compat policy. Of course, 
if something needs break, we'll discuss the case, but otherwise users who use 
this API can expect to upgrade minor releases without re-compiling their code. 
Immediate candidate is SolrJ.

I also propose that until we tag all classes, we treat "no annotation" as 
@solr.internal (except for SolrJ code). That will force us to explicitly tag 
classes that we think should at least be @solr.public, since that's the only 
annotation that should draw the attention of developers.

If people agree, we can add these annotations to the build.xml, to add proper 
text to javadocs.

Shai

On Sat, Jan 9, 2016 at 2:15 AM Jack Krupansky <jack.krupan...@gmail.com> wrote:
With the talk of 6.0 coming out real soon and not waiting for new work, will 
this 6.0/5.x issue become moot and morph into an issue for 7.0/6.x?
Settling the criteria for Solr plugin API back-compat still seems urgent, but 
if the SOLR-8475 work can quickly get committed to trunk for 6.0 maybe that 
takes some of the pressure off. Still, I'd prefer that the back-compat criteria 
be settled ASAP.
-- Jack Krupansky

On Wed, Jan 6, 2016 at 10:43 AM, Yonik Seeley <ysee...@gmail.com> wrote:
On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta <ans...@anshumgupta.net> wrote:

> As I understand, seems like there's reasonable consensus that we will:

>

> 1. provide strong back-compat for for SolrJ and REST APIs

> 2. Strive to maintain but not guarantee *strong* back-compat for Java APIs.



I think this actually represents what our current policy already is.

The sticking point is perhaps "Strive to maintain" is changing

definition to become much more lenient, to the point of being

meaningless.



Let's look at the issue that spawned this thread:

https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to

SolrIndexSearcher)



The issue is if QueryCommand and QueryResult should be moved out of

SolrIndexSearcher in 5.x (essentially a rename), or of that rename

should only be in 6.0.  If one's desire for a class rename (of classes

that are likely to be used by plugins) overrides #2, I'd argue that

means we essentially have no #2 at all.  Or perhaps I'm not grasping

why it's really that important to rename those classes.



Regarding annotations:

Multiple people have suggested annotating classes that should remain

back compat.  If we were to do this, wouldn't we want those

annotations to cover the classes in question

(SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they

cover and still be useful?



-Yonik



-

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org

For additional commands, e-mail: dev-h...@lucene.apache.org




  

Re: Breaking Java back-compat in Solr

[Shai Erera]

I think that our Maven integration only publishes Lucene/Solr artifacts to
Maven central, but I don't think the POMs actually produce the artifacts? I
could be wrong though ...

If that's the case, then we'd need to only add these to Solr's build.xml,
like it's done in Lucene.

Shai

On Sun, Jan 10, 2016 at 3:35 PM Martin Gainty <mgai...@hotmail.com> wrote:

> can you also add this to lucene pom.xml ?
>
> Martin
> __
>
>
>
>
> --
> From: ser...@gmail.com
> Date: Sun, 10 Jan 2016 07:57:26 +
> Subject: Re: Breaking Java back-compat in Solr
> To: dev@lucene.apache.org
>
>
> As for SOLR-8475, I will commit it to trunk only. But I think that we
> should come up w/ concrete annotations in our code, and annotate classes as
> we go, to back our back-compat policy. I propose that we add these:
>
> @solr.internal - this is internal API and will change without notice. No
> back-compat guarantees.
> @solr.experimental - this is a candidate for a new public API, but will
> likely change until it stabilizes. No (strong) back-compat guarantees.
> @solr.expert - while you can use this API, we cannot guarantee strong
> back-compat support. Will be on a per-case basis.
> @solr.public - this is our public API, standard back-compat policy. Of
> course, if something needs break, we'll discuss the case, but otherwise
> users who use this API can expect to upgrade minor releases without
> re-compiling their code. Immediate candidate is SolrJ.
>
> I also propose that until we tag all classes, we treat "no annotation" as
> @solr.internal (except for SolrJ code). That will force us to explicitly
> tag classes that we think should at least be @solr.public, since that's the
> only annotation that should draw the attention of developers.
>
> If people agree, we can add these annotations to the build.xml, to add
> proper text to javadocs.
>
> Shai
>
> On Sat, Jan 9, 2016 at 2:15 AM Jack Krupansky <jack.krupan...@gmail.com>
> wrote:
>
> With the talk of 6.0 coming out real soon and not waiting for new work,
> will this 6.0/5.x issue become moot and morph into an issue for 7.0/6.x?
>
> Settling the criteria for Solr plugin API back-compat still seems urgent,
> but if the SOLR-8475 work can quickly get committed to trunk for 6.0 maybe
> that takes some of the pressure off. Still, I'd prefer that the back-compat
> criteria be settled ASAP.
>
>
> -- Jack Krupansky
>
> On Wed, Jan 6, 2016 at 10:43 AM, Yonik Seeley <ysee...@gmail.com> wrote:
>
> On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta <ans...@anshumgupta.net>
> wrote:
> > As I understand, seems like there's reasonable consensus that we will:
> >
> > 1. provide strong back-compat for for SolrJ and REST APIs
> > 2. Strive to maintain but not guarantee *strong* back-compat for Java
> APIs.
>
> I think this actually represents what our current policy already is.
> The sticking point is perhaps "Strive to maintain" is changing
> definition to become much more lenient, to the point of being
> meaningless.
>
> Let's look at the issue that spawned this thread:
> https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to
> SolrIndexSearcher)
>
> The issue is if QueryCommand and QueryResult should be moved out of
> SolrIndexSearcher in 5.x (essentially a rename), or of that rename
> should only be in 6.0.  If one's desire for a class rename (of classes
> that are likely to be used by plugins) overrides #2, I'd argue that
> means we essentially have no #2 at all.  Or perhaps I'm not grasping
> why it's really that important to rename those classes.
>
> Regarding annotations:
> Multiple people have suggested annotating classes that should remain
> back compat.  If we were to do this, wouldn't we want those
> annotations to cover the classes in question
> (SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they
> cover and still be useful?
>
> -Yonik
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>

Re: Breaking Java back-compat in Solr

[Shai Erera]

As for SOLR-8475, I will commit it to trunk only. But I think that we
should come up w/ concrete annotations in our code, and annotate classes as
we go, to back our back-compat policy. I propose that we add these:

@solr.internal - this is internal API and will change without notice. No
back-compat guarantees.
@solr.experimental - this is a candidate for a new public API, but will
likely change until it stabilizes. No (strong) back-compat guarantees.
@solr.expert - while you can use this API, we cannot guarantee strong
back-compat support. Will be on a per-case basis.
@solr.public - this is our public API, standard back-compat policy. Of
course, if something needs break, we'll discuss the case, but otherwise
users who use this API can expect to upgrade minor releases without
re-compiling their code. Immediate candidate is SolrJ.

I also propose that until we tag all classes, we treat "no annotation" as
@solr.internal (except for SolrJ code). That will force us to explicitly
tag classes that we think should at least be @solr.public, since that's the
only annotation that should draw the attention of developers.

If people agree, we can add these annotations to the build.xml, to add
proper text to javadocs.

Shai

On Sat, Jan 9, 2016 at 2:15 AM Jack Krupansky 
wrote:

> With the talk of 6.0 coming out real soon and not waiting for new work,
> will this 6.0/5.x issue become moot and morph into an issue for 7.0/6.x?
>
> Settling the criteria for Solr plugin API back-compat still seems urgent,
> but if the SOLR-8475 work can quickly get committed to trunk for 6.0 maybe
> that takes some of the pressure off. Still, I'd prefer that the back-compat
> criteria be settled ASAP.
>
>
> -- Jack Krupansky
>
> On Wed, Jan 6, 2016 at 10:43 AM, Yonik Seeley  wrote:
>
>> On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta 
>> wrote:
>> > As I understand, seems like there's reasonable consensus that we will:
>> >
>> > 1. provide strong back-compat for for SolrJ and REST APIs
>> > 2. Strive to maintain but not guarantee *strong* back-compat for Java
>> APIs.
>>
>> I think this actually represents what our current policy already is.
>> The sticking point is perhaps "Strive to maintain" is changing
>> definition to become much more lenient, to the point of being
>> meaningless.
>>
>> Let's look at the issue that spawned this thread:
>> https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to
>> SolrIndexSearcher)
>>
>> The issue is if QueryCommand and QueryResult should be moved out of
>> SolrIndexSearcher in 5.x (essentially a rename), or of that rename
>> should only be in 6.0.  If one's desire for a class rename (of classes
>> that are likely to be used by plugins) overrides #2, I'd argue that
>> means we essentially have no #2 at all.  Or perhaps I'm not grasping
>> why it's really that important to rename those classes.
>>
>> Regarding annotations:
>> Multiple people have suggested annotating classes that should remain
>> back compat.  If we were to do this, wouldn't we want those
>> annotations to cover the classes in question
>> (SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they
>> cover and still be useful?
>>
>> -Yonik
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>

Re: Breaking Java back-compat in Solr

[Jack Krupansky]

With the talk of 6.0 coming out real soon and not waiting for new work,
will this 6.0/5.x issue become moot and morph into an issue for 7.0/6.x?

Settling the criteria for Solr plugin API back-compat still seems urgent,
but if the SOLR-8475 work can quickly get committed to trunk for 6.0 maybe
that takes some of the pressure off. Still, I'd prefer that the back-compat
criteria be settled ASAP.


-- Jack Krupansky

On Wed, Jan 6, 2016 at 10:43 AM, Yonik Seeley  wrote:

> On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta 
> wrote:
> > As I understand, seems like there's reasonable consensus that we will:
> >
> > 1. provide strong back-compat for for SolrJ and REST APIs
> > 2. Strive to maintain but not guarantee *strong* back-compat for Java
> APIs.
>
> I think this actually represents what our current policy already is.
> The sticking point is perhaps "Strive to maintain" is changing
> definition to become much more lenient, to the point of being
> meaningless.
>
> Let's look at the issue that spawned this thread:
> https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to
> SolrIndexSearcher)
>
> The issue is if QueryCommand and QueryResult should be moved out of
> SolrIndexSearcher in 5.x (essentially a rename), or of that rename
> should only be in 6.0.  If one's desire for a class rename (of classes
> that are likely to be used by plugins) overrides #2, I'd argue that
> means we essentially have no #2 at all.  Or perhaps I'm not grasping
> why it's really that important to rename those classes.
>
> Regarding annotations:
> Multiple people have suggested annotating classes that should remain
> back compat.  If we were to do this, wouldn't we want those
> annotations to cover the classes in question
> (SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they
> cover and still be useful?
>
> -Yonik
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Breaking Java back-compat in Solr

[Noble Paul]

Yes Anshum
HTTP APIs and SolrJ must have strong backcompat

We should have a process to mark some internal APIs as backward
compatible using some annotation/javadoc. This will help plugin
writers

On Wed, Jan 6, 2016 at 11:33 AM, Anshum Gupta  wrote:
> As I understand, seems like there's reasonable consensus that we will:
>
> 1. provide strong back-compat for for SolrJ and REST APIs
> 2. Strive to maintain but not guarantee *strong* back-compat for Java APIs.
>
> Please correct me if I'm wrong.
>
>
> On Mon, Jan 4, 2016 at 9:57 PM, Anshum Gupta  wrote:
>>
>> Hi,
>>
>> I was looking at refactoring code in Solr and it gets really tricky and
>> confusing in terms of what level of back-compat needs to be maintained.
>> Ideally, we should only maintain back-compat at the REST API level. We may
>> annotate a few really important Java APIs where we're guarantee back-compat
>> across minor versions, but we shouldn't certainly be doing that across the
>> board.
>>
>> Thoughts?
>>
>> P.S: I hope this doesn't spin-off into something I fear :)
>>
>> --
>> Anshum Gupta
>
>
>
>
> --
> Anshum Gupta



-- 
-
Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Jack Krupansky]

Solr config and schema settings should also be considered part the outer
API and require strong back-compat.

"Strive to maintain" and "simply do our best to be reasonable" certainly
seem to be good approaches to plugin-related internal APIs.

I would add "the benefits to users or committers or expert Solr
contributers are compelling" as a requirement for breaking back-compat on
any plugin-related internal API change. The problem with SOLR-8475 is that
it appears to be more of an "internal code cleanup"-only refactor that
doesn't offer any direct benefit to users or even expert Solr contributers.
That sounds more ideal for a major release trunk change and that any
backporting should be limited to maintaining plugin compatibility. At
present, nothing in the description for SOLR-8475 suggests a "compelling"
need to be in a dot release rather than a trunk/major release. If that is
not the case, it sure more be helpful if the Jira were updated to reveal
any compelling benefits beyond simply cleaning up the code.

I'm not suggesting any opposition to general code cleanup that can be done
for both trunk and the stable branch, but maybe at least it would make
sense to separate basic code cleanup from changes that may affect plugin
compatibility.

If the code cleanup dramatically simplified the implementation of a new
feature or leads to some decent performance improvement or significant
reduction in memory use, then that would be... "compelling."

I'd rather see more of a rush to get 6.0 out (and cleaned up) than a rush
to clean up code in some random dot release.

I'm +1 for any policy for dot release plugin API back-compat based on
strive/reasonable/compelling, but I'm -1 if the motivation is simply
cleanup the code at the expense of plugin compatibility and without a
compelling benefit to users or Expert contributors.

(Says the guy who has no binding vote in this matter.)



-- Jack Krupansky

On Wed, Jan 6, 2016 at 8:15 AM, Noble Paul  wrote:

> Yes Anshum
> HTTP APIs and SolrJ must have strong backcompat
>
> We should have a process to mark some internal APIs as backward
> compatible using some annotation/javadoc. This will help plugin
> writers
>
> On Wed, Jan 6, 2016 at 11:33 AM, Anshum Gupta 
> wrote:
> > As I understand, seems like there's reasonable consensus that we will:
> >
> > 1. provide strong back-compat for for SolrJ and REST APIs
> > 2. Strive to maintain but not guarantee *strong* back-compat for Java
> APIs.
> >
> > Please correct me if I'm wrong.
> >
> >
> > On Mon, Jan 4, 2016 at 9:57 PM, Anshum Gupta 
> wrote:
> >>
> >> Hi,
> >>
> >> I was looking at refactoring code in Solr and it gets really tricky and
> >> confusing in terms of what level of back-compat needs to be maintained.
> >> Ideally, we should only maintain back-compat at the REST API level. We
> may
> >> annotate a few really important Java APIs where we're guarantee
> back-compat
> >> across minor versions, but we shouldn't certainly be doing that across
> the
> >> board.
> >>
> >> Thoughts?
> >>
> >> P.S: I hope this doesn't spin-off into something I fear :)
> >>
> >> --
> >> Anshum Gupta
> >
> >
> >
> >
> > --
> > Anshum Gupta
>
>
>
> --
> -
> Noble Paul
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Breaking Java back-compat in Solr

[Yonik Seeley]

On Wed, Jan 6, 2016 at 1:03 AM, Anshum Gupta  wrote:
> As I understand, seems like there's reasonable consensus that we will:
>
> 1. provide strong back-compat for for SolrJ and REST APIs
> 2. Strive to maintain but not guarantee *strong* back-compat for Java APIs.

I think this actually represents what our current policy already is.
The sticking point is perhaps "Strive to maintain" is changing
definition to become much more lenient, to the point of being
meaningless.

Let's look at the issue that spawned this thread:
https://issues.apache.org/jira/browse/SOLR-8475  (Some refactoring to
SolrIndexSearcher)

The issue is if QueryCommand and QueryResult should be moved out of
SolrIndexSearcher in 5.x (essentially a rename), or of that rename
should only be in 6.0.  If one's desire for a class rename (of classes
that are likely to be used by plugins) overrides #2, I'd argue that
means we essentially have no #2 at all.  Or perhaps I'm not grasping
why it's really that important to rename those classes.

Regarding annotations:
Multiple people have suggested annotating classes that should remain
back compat.  If we were to do this, wouldn't we want those
annotations to cover the classes in question
(SolrIndexSearcher,QueryCommand,QueryResult)?  If not, what would they
cover and still be useful?

-Yonik

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Mark Miller]

We have never promised or delivered back compat on Java APIs beyond best
effort. I agree we should stick to http APIs and solrj for stronger back
compat, and simply do our best to be reasonable with the rest.

Mark
On Tue, Jan 5, 2016 at 2:05 AM Shai Erera  wrote:

> In Lucene there are three types of public APIs:
>
> "public stable": not annotated and for them we try to maintain backward
> compatibility across minor releases
> "public experimental": annotated with @lucene.experimental. If possible,
> back-compat is maintained, but otherwise you can expect them to change
> between minor versions
> "public internal": annotated with @lucene.internal. Due to Java
> limitations, these are public classes/methods, but we have no intention to
> keep them backward compatible.
>
> Even the "public stable" APIs are further divided, though not through
> annotations, and more a feel and consensus on a particular issue, into
> expert and non-expert APIs. Take for example the overhaul that was done to
> SpanQuery in LUCENE-6308. I believe it was decided that users who implement
> their own SpanQuery, are probably expert users and can take the API changes.
>
> When it comes to Solr, I always viewed user-facing APIs as SolrJ and the
> REST APIs and therefore referred to them as "public stable". When it comes
> to plug-ins, I view them at most as "public expert". I don't think that
> Solr users who write their own plug-ins should be viewed as simple users.
> They are more than the average user IMO. Even if the plug-in that they
> write is super simple, like adding a log message, how bad can an API break
> be for them to change their code?
>
> I think that, even if not publicly announced, in Lucene we sort of dropped
> the requirement of jar drop-in ability. I personally believe that there are
> only few users who expect to upgrade a search server running Solr from
> version X to Y, without this affecting their code. So for the majority of
> users who use SolrJ and the REST APIs, this will be the case since we
> guarantee back-compat. For those who write their own plug-ins, it may be a
> matter of luck. Either they do or don't have to recompile their code. For
> the super expert ones, e.g. who write a plug-in from scratch or do really
> invasive things, I think it's fair to ask them to align with the new APIs.
>
> I implemented a bunch of SpanQuery who worked fine in 5.2.1. Last week
> I've decided to upgrade to 5.4 and had to rewrite the queries. In the
> process, I discovered that the new API helps me implement my queries
> better, as well one of the queries was now available from Lucene. I didn't
> complain about it, as painful as it was, since I trust the developers who
> have made that decision to weigh the pros and cons of the API changes.
>
> Also, I find it odd that some of the arguments made here distinguish Solr
> users from Lucene users. Ain't a Solr user who implements his own
> QParsePlugin subject to the same API changes (e.g. SpanQuery) in Lucene,
> that non-Solr users are? Why do we even attempt to make that distinction?
> Solr is built on top of Lucene, so most likely the plugin that you write
> will need to interact w/ Lucene API too.
>
> As to the particular issue in SOLR-8475, the API break will be resolved by
> any modern IDE with a simple "Organize Imports". And I don't view
> SolrIndexSearcher as anywhere near the top-level of the APIs that we should
> maintain. Nor do I think that SolrCache is one such (as another example,
> unrelated to that issue). The interfaces that define plugins, maybe. But
> the internals are subject, IMO, to either be "solr expert" or "solr
> internal".
>
> If we want to refactor those pieces in Solr (and I think that we should,
> and I try to help with it), we must be more flexible around our API
> guarantees. Also, trust the devs to not rewrite APIs just for the hack of
> it, but if they do, it's for a good reason, for them (as code maintainers)
> and our users.
>
> Shai
>
> On Tue, Jan 5, 2016 at 8:36 AM Noble Paul  wrote:
>
>> I would say, SolrJ and REST APIs MUST BE backward  compatible between
>> minor versions.
>>
>> The question is about the internal java APIs. It is impossible to get
>> 100% right on these things. If we can start annotating classes/methods
>> and let users suggest stuff then we should be in a reasonably good
>> situation over an year.
>> As a first step open a ticket and define a process to make a certain
>> API backcompat .
>> may be an annotation or whatever.
>>
>> On Tue, Jan 5, 2016 at 11:19 AM, Anshum Gupta 
>> wrote:
>> > Thanks David,
>> >
>> > I agree with what you've suggested but the bigger question here again is
>> > *which* files do we guarantee back-compat for. I suggest we guarantee
>> > back-compat for SolrJ and REST APIs. For everything else i.e. Java
>> APIs, we
>> > should try and maintain back-compat but there shouldn't be a guarantee
>> and
>> > should be the 

Re: Breaking Java back-compat in Solr

[Anshum Gupta]

As I understand, seems like there's reasonable consensus that we will:

1. provide strong back-compat for for SolrJ and REST APIs
2. Strive to maintain but not guarantee *strong* back-compat for Java APIs.

Please correct me if I'm wrong.


On Mon, Jan 4, 2016 at 9:57 PM, Anshum Gupta  wrote:

> Hi,
>
> I was looking at refactoring code in Solr and it gets really tricky and
> confusing in terms of what level of back-compat needs to be maintained.
> Ideally, we should only maintain back-compat at the REST API level. We may
> annotate a few really important Java APIs where we're guarantee back-compat
> across minor versions, but we shouldn't certainly be doing that across the
> board.
>
> Thoughts?
>
> P.S: I hope this doesn't spin-off into something I fear :)
>
> --
> Anshum Gupta
>



-- 
Anshum Gupta

Re: Breaking Java back-compat in Solr

[Jack Krupansky]

I suspect that half the issue here is that 6.0 is viewed as too far away so
that any trunk-only enhancements are then seen as not having any near-term
relevance. If 6.0 were targeted for sometime within the next six months,
would that not take a lot out of the urgency for major/breaking changes in
dot releases?

Anybody object to a Solr 6.0 in June or thereabouts? Would the folks in
Elasticsearch land object to a Lucene 6.0 release in that timeframe (if not
sooner!)?

I'm +1 for saying that dot releases be limited to "no surprises", easy
upgrades, with no app/custom code changes for the external and general
internal APIs, but under the condition that a major release is never more
than a year away. In any case, make a commitment to users that they can
always safely and painlessly upgrade from x.y to x.z without code changes.

Sure, minor and even major enhancements can occur in dot releases - to the
extent that they "drop in" without introducing compatibility issues, with
compatibility defined as back-compat with the Lucene index, the HTTP API,
the Solr plugin API and any general core interfaces that reasonable plugins
might use.

And if this policy puts greater pressure on getting an earlier 6.0 release,
so be it. +1 for that.

Whether the Lucene guys have the same concerns as the Solr guys is an
interesting question.


-- Jack Krupansky

On Mon, Jan 4, 2016 at 12:30 PM, Yonik Seeley  wrote:

> On Mon, Jan 4, 2016 at 12:07 PM, Alexandre Rafalovitch
>  wrote:
> > Solr plugin story is muddy enough as it is. Plugins are hard to find,
> > share. So, in my eyes, breaking them is not a big effect as if we had
> > a big active registry.
>
> I think private plugins / components are more the issue here (a custom
> qparser, search component, update processor).
> The basic question is: should people using these be able to upgrade
> from 5.4 to 5.5 without having to change and recompile their code?
>
> -Yonik
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Breaking Java back-compat in Solr

[Gregory Chanan]

Has there been any discussion about annotating back compat expectations
universally, similar to hadoop's use of InterfaceStability?  That of course
only solves the first issue: "gets really tricky and confusing in terms of
what level of back-compat needs to be maintained", because it's defined by
the annotation.  It doesn't solve the policy issue of which annotation to
use for a given class, of course.

On Mon, Jan 4, 2016 at 12:55 PM, Jack Krupansky 
wrote:

> I suspect that half the issue here is that 6.0 is viewed as too far away
> so that any trunk-only enhancements are then seen as not having any
> near-term relevance. If 6.0 were targeted for sometime within the next six
> months, would that not take a lot out of the urgency for major/breaking
> changes in dot releases?
>
> Anybody object to a Solr 6.0 in June or thereabouts? Would the folks in
> Elasticsearch land object to a Lucene 6.0 release in that timeframe (if not
> sooner!)?
>
> I'm +1 for saying that dot releases be limited to "no surprises", easy
> upgrades, with no app/custom code changes for the external and general
> internal APIs, but under the condition that a major release is never more
> than a year away. In any case, make a commitment to users that they can
> always safely and painlessly upgrade from x.y to x.z without code changes.
>
> Sure, minor and even major enhancements can occur in dot releases - to the
> extent that they "drop in" without introducing compatibility issues, with
> compatibility defined as back-compat with the Lucene index, the HTTP API,
> the Solr plugin API and any general core interfaces that reasonable plugins
> might use.
>
> And if this policy puts greater pressure on getting an earlier 6.0
> release, so be it. +1 for that.
>
> Whether the Lucene guys have the same concerns as the Solr guys is an
> interesting question.
>
>
> -- Jack Krupansky
>
> On Mon, Jan 4, 2016 at 12:30 PM, Yonik Seeley  wrote:
>
>> On Mon, Jan 4, 2016 at 12:07 PM, Alexandre Rafalovitch
>>  wrote:
>> > Solr plugin story is muddy enough as it is. Plugins are hard to find,
>> > share. So, in my eyes, breaking them is not a big effect as if we had
>> > a big active registry.
>>
>> I think private plugins / components are more the issue here (a custom
>> qparser, search component, update processor).
>> The basic question is: should people using these be able to upgrade
>> from 5.4 to 5.5 without having to change and recompile their code?
>>
>> -Yonik
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>

Re: Breaking Java back-compat in Solr

[Shawn Heisey]

On 1/4/2016 1:55 PM, Jack Krupansky wrote:
> I suspect that half the issue here is that 6.0 is viewed as too far
> away so that any trunk-only enhancements are then seen as not having
> any near-term relevance. If 6.0 were targeted for sometime within the
> next six months, would that not take a lot out of the urgency for
> major/breaking changes in dot releases?
>
> Anybody object to a Solr 6.0 in June or thereabouts? Would the folks
> in Elasticsearch land object to a Lucene 6.0 release in that timeframe
> (if not sooner!)?
>

I said much the same thing on SOLR-8475 a short time ago.  I'm all for
creating branch_6x in the very near future and looking forward to the
actual release a few months after that.  The CHANGES.txt for 5.5 looks
very extensive for both solr and lucene, so I believe that we should
probably get 5.5 out the door first.

> I'm +1 for saying that dot releases be limited to "no surprises", easy
> upgrades, with no app/custom code changes for the external and general
> internal APIs, but under the condition that a major release is never
> more than a year away. In any case, make a commitment to users that
> they can always safely and painlessly upgrade from x.y to x.z without
> code changes.

That is what we aim for.  I personally am not opposed to making very
minor changes to my custom code when a new minor version comes out, but
if that's avoidable, everybody wins.

Any time I write code that uses the Solr API (separate from the SolrJ
API), I presume ahead of time that my plugin jar may not work when I
upgrade Solr.  This is probably paranoia, but because I recompile my
plugin anytime I upgrade, I know that any problems I encounter are
likely due to my own mistakes.

> Whether the Lucene guys have the same concerns as the Solr guys is an
> interesting question.

A "user" of Lucene is typically a developer, someone who presumably
knows how to fix problems created by API changes.  If their code stops
working when they upgrade a dependency like Lucene, they can adapt. 
Also, because of the very nature of the project, I think that Lucene
devs are very good about indicating which Lucene APIs are
expert/internal and subject to change.  Lucene internals are very
complex, but we have some incredibly smart people here who know them
very well, and they know which APIs are unlikely to be found in typical
user programs.

A typical Solr user is not a developer, and just wants everything to
work, potentially with custom code that they cannot change or
recompile.  I don't think the Solr devs are less intelligent than the
Lucene devs ... but because Solr is primarily an application rather than
an API, I don't think that there is as much effort in the Solr code to
indicate which APIs should be considered expert/internal.

I know from experience that third-party Solr plugins are sometimes
extremely version-specific, so the goal of custom code working with a
new minor version is not always achieved.

Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[david.w.smi...@gmail.com]

Great topic Anshum.

I’ve been frustrated with the back-compat situation since Lucene/Solr 5 as
a maintainer of my “SolrTextTagger”.  One version of the plugin supports
4.3 thru the end of the 4x line, whereas on the 5x side I’ve needed 3
versions already (5.0-5,1, 5.2, 5.3+)!  Sometimes on the API consumer side
there is simply a new way compatible with the old way and those don’t
bother me much; what’s annoying is just flat-out differences that can’t
easily be coded around without reflection.  But in my experience through
the SolrTextTagger (YMMV), Solr hasn’t been the main offender of such
changes — it’s Lucene changes (be it removing liveDocs bits to get
postings, changing the BytesRefAttribute signature, or something I don’t
remember now). At least twice it was avoidable IMO.  Nevertheless, we Solr
devs should come up with a back-compat policy, a simple document/paragraph
perhaps, and save it somewhere so we can refer to it.  Lets not have to dig
through the mailing list to know our policy some day in the future when we
want to explain it!

I suggest that Solr's *default* policy for any source file (Java API) that
doesn’t otherwise annotate a back-compat statement is to be permissive to
changes — developer judgement on how much back-compat makes sense to them.
I say this because the Solr code base is large and I think a relatively
small portion of it should aim for stability.  Lets take SearchComponent as
an example.  *That* needs to be stable.  But does HighlightComponent?  I
really don’t think so; besides, it only has one overridable method defined
by this class that isn’t inherited.   Oddly (IMO) there is a separate
abstraction SolrHighlighter and I can intuit that it’s this guy that was
intended to be the abstraction of the Highlighter implementation, not the
some-what generic HighlightComponent.  So arguably SolrHighlighter should
be stable.  DefaultSolrHighlighter is debatable as being stable — it’s a
specific highlighter but it has a bunch of methods designed to be
overridden (and I have done so).  So I think that’s a judgement call
(developer prerogative).

Should we apply a back-compat policy statement (either through a simple
comment or better through a new annotation), I don’t think we should feel
helpless to strictly abide by it for the entire major version range.  We
might decide that such changes are possible provided it gets at least
one +1 and no -1 veto from another developer.

Summary:
* Publish a back-compat policy/approach where we can refer to it easily.
* The default policy of source files without annotations is the developer’s
prerogative — no back-compat.
* Annotate the back-compat Java source files as-such and allow us to break
back-compat only if voted.

~ David

On Mon, Jan 4, 2016 at 11:28 AM Anshum Gupta  wrote:

> Hi,
>
> I was looking at refactoring code in Solr and it gets really tricky and
> confusing in terms of what level of back-compat needs to be maintained.
> Ideally, we should only maintain back-compat at the REST API level. We may
> annotate a few really important Java APIs where we're guarantee back-compat
> across minor versions, but we shouldn't certainly be doing that across the
> board.
>
> Thoughts?
>
> P.S: I hope this doesn't spin-off into something I fear :)
>
> --
> Anshum Gupta
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Breaking Java back-compat in Solr

[Shai Erera]

In Lucene there are three types of public APIs:

"public stable": not annotated and for them we try to maintain backward
compatibility across minor releases
"public experimental": annotated with @lucene.experimental. If possible,
back-compat is maintained, but otherwise you can expect them to change
between minor versions
"public internal": annotated with @lucene.internal. Due to Java
limitations, these are public classes/methods, but we have no intention to
keep them backward compatible.

Even the "public stable" APIs are further divided, though not through
annotations, and more a feel and consensus on a particular issue, into
expert and non-expert APIs. Take for example the overhaul that was done to
SpanQuery in LUCENE-6308. I believe it was decided that users who implement
their own SpanQuery, are probably expert users and can take the API changes.

When it comes to Solr, I always viewed user-facing APIs as SolrJ and the
REST APIs and therefore referred to them as "public stable". When it comes
to plug-ins, I view them at most as "public expert". I don't think that
Solr users who write their own plug-ins should be viewed as simple users.
They are more than the average user IMO. Even if the plug-in that they
write is super simple, like adding a log message, how bad can an API break
be for them to change their code?

I think that, even if not publicly announced, in Lucene we sort of dropped
the requirement of jar drop-in ability. I personally believe that there are
only few users who expect to upgrade a search server running Solr from
version X to Y, without this affecting their code. So for the majority of
users who use SolrJ and the REST APIs, this will be the case since we
guarantee back-compat. For those who write their own plug-ins, it may be a
matter of luck. Either they do or don't have to recompile their code. For
the super expert ones, e.g. who write a plug-in from scratch or do really
invasive things, I think it's fair to ask them to align with the new APIs.

I implemented a bunch of SpanQuery who worked fine in 5.2.1. Last week I've
decided to upgrade to 5.4 and had to rewrite the queries. In the process, I
discovered that the new API helps me implement my queries better, as well
one of the queries was now available from Lucene. I didn't complain about
it, as painful as it was, since I trust the developers who have made that
decision to weigh the pros and cons of the API changes.

Also, I find it odd that some of the arguments made here distinguish Solr
users from Lucene users. Ain't a Solr user who implements his own
QParsePlugin subject to the same API changes (e.g. SpanQuery) in Lucene,
that non-Solr users are? Why do we even attempt to make that distinction?
Solr is built on top of Lucene, so most likely the plugin that you write
will need to interact w/ Lucene API too.

As to the particular issue in SOLR-8475, the API break will be resolved by
any modern IDE with a simple "Organize Imports". And I don't view
SolrIndexSearcher as anywhere near the top-level of the APIs that we should
maintain. Nor do I think that SolrCache is one such (as another example,
unrelated to that issue). The interfaces that define plugins, maybe. But
the internals are subject, IMO, to either be "solr expert" or "solr
internal".

If we want to refactor those pieces in Solr (and I think that we should,
and I try to help with it), we must be more flexible around our API
guarantees. Also, trust the devs to not rewrite APIs just for the hack of
it, but if they do, it's for a good reason, for them (as code maintainers)
and our users.

Shai

On Tue, Jan 5, 2016 at 8:36 AM Noble Paul  wrote:

> I would say, SolrJ and REST APIs MUST BE backward  compatible between
> minor versions.
>
> The question is about the internal java APIs. It is impossible to get
> 100% right on these things. If we can start annotating classes/methods
> and let users suggest stuff then we should be in a reasonably good
> situation over an year.
> As a first step open a ticket and define a process to make a certain
> API backcompat .
> may be an annotation or whatever.
>
> On Tue, Jan 5, 2016 at 11:19 AM, Anshum Gupta 
> wrote:
> > Thanks David,
> >
> > I agree with what you've suggested but the bigger question here again is
> > *which* files do we guarantee back-compat for. I suggest we guarantee
> > back-compat for SolrJ and REST APIs. For everything else i.e. Java APIs,
> we
> > should try and maintain back-compat but there shouldn't be a guarantee
> and
> > should be the developer's prerogative else defining what is to be
> annotated
> > as "Long term support" and what isn't becomes subject to usage and
> debate.
> >
> > On Tue, Jan 5, 2016 at 10:32 AM, david.w.smi...@gmail.com
> >  wrote:
> >>
> >> Great topic Anshum.
> >>
> >> I’ve been frustrated with the back-compat situation since Lucene/Solr 5
> as
> >> a maintainer of my “SolrTextTagger”.  One version of the plugin
> 

Re: Breaking Java back-compat in Solr

[Noble Paul]

I would say, SolrJ and REST APIs MUST BE backward  compatible between
minor versions.

The question is about the internal java APIs. It is impossible to get
100% right on these things. If we can start annotating classes/methods
and let users suggest stuff then we should be in a reasonably good
situation over an year.
As a first step open a ticket and define a process to make a certain
API backcompat .
may be an annotation or whatever.

On Tue, Jan 5, 2016 at 11:19 AM, Anshum Gupta  wrote:
> Thanks David,
>
> I agree with what you've suggested but the bigger question here again is
> *which* files do we guarantee back-compat for. I suggest we guarantee
> back-compat for SolrJ and REST APIs. For everything else i.e. Java APIs, we
> should try and maintain back-compat but there shouldn't be a guarantee and
> should be the developer's prerogative else defining what is to be annotated
> as "Long term support" and what isn't becomes subject to usage and debate.
>
> On Tue, Jan 5, 2016 at 10:32 AM, david.w.smi...@gmail.com
>  wrote:
>>
>> Great topic Anshum.
>>
>> I’ve been frustrated with the back-compat situation since Lucene/Solr 5 as
>> a maintainer of my “SolrTextTagger”.  One version of the plugin supports 4.3
>> thru the end of the 4x line, whereas on the 5x side I’ve needed 3 versions
>> already (5.0-5,1, 5.2, 5.3+)!  Sometimes on the API consumer side there is
>> simply a new way compatible with the old way and those don’t bother me much;
>> what’s annoying is just flat-out differences that can’t easily be coded
>> around without reflection.  But in my experience through the SolrTextTagger
>> (YMMV), Solr hasn’t been the main offender of such changes — it’s Lucene
>> changes (be it removing liveDocs bits to get postings, changing the
>> BytesRefAttribute signature, or something I don’t remember now). At least
>> twice it was avoidable IMO.  Nevertheless, we Solr devs should come up with
>> a back-compat policy, a simple document/paragraph perhaps, and save it
>> somewhere so we can refer to it.  Lets not have to dig through the mailing
>> list to know our policy some day in the future when we want to explain it!
>>
>> I suggest that Solr's *default* policy for any source file (Java API) that
>> doesn’t otherwise annotate a back-compat statement is to be permissive to
>> changes — developer judgement on how much back-compat makes sense to them.
>> I say this because the Solr code base is large and I think a relatively
>> small portion of it should aim for stability.  Lets take SearchComponent as
>> an example.  *That* needs to be stable.  But does HighlightComponent?  I
>> really don’t think so; besides, it only has one overridable method defined
>> by this class that isn’t inherited.   Oddly (IMO) there is a separate
>> abstraction SolrHighlighter and I can intuit that it’s this guy that was
>> intended to be the abstraction of the Highlighter implementation, not the
>> some-what generic HighlightComponent.  So arguably SolrHighlighter should be
>> stable.  DefaultSolrHighlighter is debatable as being stable — it’s a
>> specific highlighter but it has a bunch of methods designed to be overridden
>> (and I have done so).  So I think that’s a judgement call (developer
>> prerogative).
>>
>> Should we apply a back-compat policy statement (either through a simple
>> comment or better through a new annotation), I don’t think we should feel
>> helpless to strictly abide by it for the entire major version range.  We
>> might decide that such changes are possible provided it gets at least one +1
>> and no -1 veto from another developer.
>>
>> Summary:
>> * Publish a back-compat policy/approach where we can refer to it easily.
>> * The default policy of source files without annotations is the
>> developer’s prerogative — no back-compat.
>> * Annotate the back-compat Java source files as-such and allow us to break
>> back-compat only if voted.
>>
>> ~ David
>>
>> On Mon, Jan 4, 2016 at 11:28 AM Anshum Gupta 
>> wrote:
>>>
>>> Hi,
>>>
>>> I was looking at refactoring code in Solr and it gets really tricky and
>>> confusing in terms of what level of back-compat needs to be maintained.
>>> Ideally, we should only maintain back-compat at the REST API level. We may
>>> annotate a few really important Java APIs where we're guarantee back-compat
>>> across minor versions, but we shouldn't certainly be doing that across the
>>> board.
>>>
>>> Thoughts?
>>>
>>> P.S: I hope this doesn't spin-off into something I fear :)
>>>
>>> --
>>> Anshum Gupta
>>
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>> http://www.solrenterprisesearchserver.com
>
>
>
>
> --
> Anshum Gupta



-- 
-
Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Anshum Gupta]

Thanks David,

I agree with what you've suggested but the bigger question here again is
*which* files do we guarantee back-compat for. I suggest we guarantee
back-compat for SolrJ and REST APIs. For everything else i.e. Java APIs, we
should try and maintain back-compat but there shouldn't be a guarantee and
should be the developer's prerogative else defining what is to be annotated
as "Long term support" and what isn't becomes subject to usage and debate.

On Tue, Jan 5, 2016 at 10:32 AM, david.w.smi...@gmail.com <
david.w.smi...@gmail.com> wrote:

> Great topic Anshum.
>
> I’ve been frustrated with the back-compat situation since Lucene/Solr 5 as
> a maintainer of my “SolrTextTagger”.  One version of the plugin supports
> 4.3 thru the end of the 4x line, whereas on the 5x side I’ve needed 3
> versions already (5.0-5,1, 5.2, 5.3+)!  Sometimes on the API consumer side
> there is simply a new way compatible with the old way and those don’t
> bother me much; what’s annoying is just flat-out differences that can’t
> easily be coded around without reflection.  But in my experience through
> the SolrTextTagger (YMMV), Solr hasn’t been the main offender of such
> changes — it’s Lucene changes (be it removing liveDocs bits to get
> postings, changing the BytesRefAttribute signature, or something I don’t
> remember now). At least twice it was avoidable IMO.  Nevertheless, we Solr
> devs should come up with a back-compat policy, a simple document/paragraph
> perhaps, and save it somewhere so we can refer to it.  Lets not have to dig
> through the mailing list to know our policy some day in the future when we
> want to explain it!
>
> I suggest that Solr's *default* policy for any source file (Java API) that
> doesn’t otherwise annotate a back-compat statement is to be permissive to
> changes — developer judgement on how much back-compat makes sense to them.
> I say this because the Solr code base is large and I think a relatively
> small portion of it should aim for stability.  Lets take SearchComponent as
> an example.  *That* needs to be stable.  But does HighlightComponent?  I
> really don’t think so; besides, it only has one overridable method defined
> by this class that isn’t inherited.   Oddly (IMO) there is a separate
> abstraction SolrHighlighter and I can intuit that it’s this guy that was
> intended to be the abstraction of the Highlighter implementation, not the
> some-what generic HighlightComponent.  So arguably SolrHighlighter should
> be stable.  DefaultSolrHighlighter is debatable as being stable — it’s a
> specific highlighter but it has a bunch of methods designed to be
> overridden (and I have done so).  So I think that’s a judgement call
> (developer prerogative).
>
> Should we apply a back-compat policy statement (either through a simple
> comment or better through a new annotation), I don’t think we should feel
> helpless to strictly abide by it for the entire major version range.  We
> might decide that such changes are possible provided it gets at least
> one +1 and no -1 veto from another developer.
>
> Summary:
> * Publish a back-compat policy/approach where we can refer to it easily.
> * The default policy of source files without annotations is the
> developer’s prerogative — no back-compat.
> * Annotate the back-compat Java source files as-such and allow us to break
> back-compat only if voted.
>
> ~ David
>
> On Mon, Jan 4, 2016 at 11:28 AM Anshum Gupta 
> wrote:
>
>> Hi,
>>
>> I was looking at refactoring code in Solr and it gets really tricky and
>> confusing in terms of what level of back-compat needs to be maintained.
>> Ideally, we should only maintain back-compat at the REST API level. We may
>> annotate a few really important Java APIs where we're guarantee back-compat
>> across minor versions, but we shouldn't certainly be doing that across the
>> board.
>>
>> Thoughts?
>>
>> P.S: I hope this doesn't spin-off into something I fear :)
>>
>> --
>> Anshum Gupta
>>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>



-- 
Anshum Gupta

Re: Breaking Java back-compat in Solr

[Upayavira]

Add in major components, such as SearchComponent, IpdateProcessor, etc,
they need to be considered stable.


On Tue, Jan 5, 2016, at 05:49 AM, Anshum Gupta wrote:
> Thanks David,
>
> I agree with what you've suggested but the bigger question here again
> is *which* files a we guarantee back-compat for. I suggest we
> guarantee back-compat for SolrJ and REST APIs. For everything else
> i.e. Java APIs, we should try and maintain back-compat but there
> shouldn't be a guarantee and should be the developer's prerogative
> else defining what is to be annotated as "Long term support" and what
> isn't becomes subject to usage and debate.
>
> On Tue, Jan 5, 2016 at 10:32 AM, david.w.smi...@gmail.com
>  wrote:
>> Great topic Anshum.
>>
>> I’ve been frustrated with the back-compat situation since Lucene/Solr
>> 5 as a maintainer of my “SolrTextTagger”.  One version of the plugin
>> supports 4.3 thru the end of the 4x line, whereas on the 5x side I’ve
>> needed 3 versions already (5.0-5,1, 5.2, 5.3+)!  Sometimes on the API
>> consumer side there is simply a new way compatible with the old way
>> and those don’t bother me much; what’s annoying is just flat-out
>> differences that can’t easily be coded around without reflection.
>> But in my experience through the SolrTextTagger (YMMV), Solr hasn’t
>> been the main offender of such changes — it’s Lucene changes (be it
>> removing liveDocs bits to get postings, changing the
>> BytesRefAttribute signature, or something I don’t remember now). At
>> least twice it was avoidable IMO.  Nevertheless, we Solr devs should
>> come up with a back-compat policy, a simple document/paragraph
>> perhaps, and save it somewhere so we can refer to it.  Lets not have
>> to dig through the mailing list to know our policy some day in the
>> future when we want to explain it!
>>
>> I suggest that Solr's *default* policy for any source file (Java API)
>> that doesn’t otherwise annotate a back-compat statement is to be
>> permissive to changes — developer judgement on how much back-compat
>> makes sense to them.  I say this because the Solr code base is large
>> and I think a relatively small portion of it should aim for
>> stability.  Lets take SearchComponent as an example.  *That* needs to
>> be stable.  But does HighlightComponent?  I really don’t think so;
>> besides, it only has one overridable method defined by this class
>> that isn’t inherited.   Oddly (IMO) there is a separate abstraction
>> SolrHighlighter and I can intuit that it’s this guy that was intended
>> to be the abstraction of the Highlighter implementation, not the some-
>> what generic HighlightComponent.  So arguably SolrHighlighter should
>> be stable.  DefaultSolrHighlighter is debatable as being stable —
>> it’s a specific highlighter but it has a bunch of methods designed to
>> be overridden (and I have done so).  So I think that’s a judgement
>> call (developer prerogative).
>>
>> Should we apply a back-compat policy statement (either through a
>> simple comment or better through a new annotation), I don’t think we
>> should feel helpless to strictly abide by it for the entire major
>> version range.  We might decide that such changes are possible
>> provided it gets at least one +1 and no -1 veto from another
>> developer.
>>
>> Summary:
>> * Publish a back-compat policy/approach where we can refer to it
>>   easily.
>> * The default policy of source files without annotations is the
>>   developer’s prerogative — no back-compat.
>> * Annotate the back-compat Java source files as-such and allow us to
>>   break back-compat only if voted.
>>
>> ~ David
>>
>> On Mon, Jan 4, 2016 at 11:28 AM Anshum Gupta
>>  wrote:
>>> Hi,
>>>
>>> I was looking at refactoring code in Solr and it gets really tricky
>>> and confusing in terms of what level of back-compat needs to be
>>> maintained. Ideally, we should only maintain back-compat at the REST
>>> API level. We may annotate a few really important Java APIs where
>>> we're guarantee back-compat across minor versions, but we shouldn't
>>> certainly be doing that across the board.
>>>
>>> Thoughts?
>>>
>>> P.S: I hope this doesn't spin-off into something I fear :)
>>>
>>> --
>>> Anshum Gupta
>>
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>> http://www.solrenterprisesearchserver.com
>>
>
>
>
> --
> Anshum Gupta

Re: Breaking Java back-compat in Solr

[Yonik Seeley]

On Mon, Jan 4, 2016 at 12:07 PM, Alexandre Rafalovitch
 wrote:
> Solr plugin story is muddy enough as it is. Plugins are hard to find,
> share. So, in my eyes, breaking them is not a big effect as if we had
> a big active registry.

I think private plugins / components are more the issue here (a custom
qparser, search component, update processor).
The basic question is: should people using these be able to upgrade
from 5.4 to 5.5 without having to change and recompile their code?

-Yonik

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Erik Hatcher]

Anshum -

Interesting timing in bringing this up.  I’ve just been bitten by the 
ShardHandler API change from https://issues.apache.org/jira/browse/SOLR-7714 
 - I’m working on adapting 
some code from 
https://github.com/LucidWorks/query-autofiltering-component/tree/master/solr5.x 

 which was using Solr 5.1 API and now it breaks.  I’m in the midst of adapting 
the build and code, so I’ve locally gotten past the change that broke this API 
usage: 
https://github.com/LucidWorks/query-autofiltering-component/blob/master/solr5.x/src/main/java/org/apache/solr/handler/component/QueryAutoFilteringComponent.java#L807
 

 where s/checkDistributed/prepDistributed/ was required.  That method name 
change was probably best on trunk-only?[though, maybe there’s other ways to 
do what that QueryAutoFilteringComponent should be doing in distributed mode, 
which I’m looking into as well]

Erik





> On Jan 4, 2016, at 11:27 AM, Anshum Gupta  wrote:
> 
> Hi,
> 
> I was looking at refactoring code in Solr and it gets really tricky and 
> confusing in terms of what level of back-compat needs to be maintained. 
> Ideally, we should only maintain back-compat at the REST API level. We may 
> annotate a few really important Java APIs where we're guarantee back-compat 
> across minor versions, but we shouldn't certainly be doing that across the 
> board.
> 
> Thoughts?
> 
> P.S: I hope this doesn't spin-off into something I fear :)
> 
> -- 
> Anshum Gupta

Re: Breaking Java back-compat in Solr

[Noble Paul]

IMHO
It's OK if the user has to go through recompile/modify their custom
component as long as it is obvious as to what is changed and we give
clear documentation on how to upgrade

On Mon, Jan 4, 2016 at 11:00 PM, Yonik Seeley  wrote:
> On Mon, Jan 4, 2016 at 12:07 PM, Alexandre Rafalovitch
>  wrote:
>> Solr plugin story is muddy enough as it is. Plugins are hard to find,
>> share. So, in my eyes, breaking them is not a big effect as if we had
>> a big active registry.
>
> I think private plugins / components are more the issue here (a custom
> qparser, search component, update processor).
> The basic question is: should people using these be able to upgrade
> from 5.4 to 5.5 without having to change and recompile their code?
>
> -Yonik
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>



-- 
-
Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Jack Krupansky]

Probably back-compat for the Solr plugin API as well - people have custom
plugins that should work without a needed refactor. That would include
custom tokenizers, token filters, char filters, query parsers, request
handlers.

That probably implies back-compat for the major APIs for Solr core as well
- the code that plugins need to call.


-- Jack Krupansky

On Mon, Jan 4, 2016 at 11:27 AM, Anshum Gupta 
wrote:

> Hi,
>
> I was looking at refactoring code in Solr and it gets really tricky and
> confusing in terms of what level of back-compat needs to be maintained.
> Ideally, we should only maintain back-compat at the REST API level. We may
> annotate a few really important Java APIs where we're guarantee back-compat
> across minor versions, but we shouldn't certainly be doing that across the
> board.
>
> Thoughts?
>
> P.S: I hope this doesn't spin-off into something I fear :)
>
> --
> Anshum Gupta
>

Re: Breaking Java back-compat in Solr

[Yonik Seeley]

On Mon, Jan 4, 2016 at 11:35 AM, Jack Krupansky
 wrote:
> Probably back-compat for the Solr plugin API as well - people have custom
> plugins that should work without a needed refactor. That would include
> custom tokenizers, token filters, char filters, query parsers, request
> handlers.
>
> That probably implies back-compat for the major APIs for Solr core as well -
> the code that plugins need to call.

That reflects our current policy (i.e. things like
SolrIndexSearcher/QueryCommand/QueryResult would be covered).

I think Anshum is looking for something that would allow the refactors
in this issue to be committed to 5.5
https://issues.apache.org/jira/browse/SOLR-8475

-Yonik

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Anshum Gupta]

Yes, I'm looking at a long term consensus on issues like SOLR-8475.

It would make it possible to improve the code without having to wait until
a major version is released.

On Mon, Jan 4, 2016 at 10:13 PM, Yonik Seeley  wrote:

> On Mon, Jan 4, 2016 at 11:35 AM, Jack Krupansky
>  wrote:
> > Probably back-compat for the Solr plugin API as well - people have custom
> > plugins that should work without a needed refactor. That would include
> > custom tokenizers, token filters, char filters, query parsers, request
> > handlers.
> >
> > That probably implies back-compat for the major APIs for Solr core as
> well -
> > the code that plugins need to call.
>
> That reflects our current policy (i.e. things like
> SolrIndexSearcher/QueryCommand/QueryResult would be covered).
>
> I think Anshum is looking for something that would allow the refactors
> in this issue to be committed to 5.5
> https://issues.apache.org/jira/browse/SOLR-8475
>
> -Yonik
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Anshum Gupta

Re: Breaking Java back-compat in Solr

[Noble Paul]

The fact that users can write plugins and the plugins can access
almost any public methods make it extremely hard to remain backward
compatible. However , we should not hesitate to break backcompat (if
required). Users have the choice of sticking to the old version and
rewrite his plugins using the nw API.
However public/REST APIs should not break backcompat in a minor verion

On Mon, Jan 4, 2016 at 10:17 PM, Anshum Gupta  wrote:
> Yes, I'm looking at a long term consensus on issues like SOLR-8475.
>
> It would make it possible to improve the code without having to wait until a
> major version is released.
>
> On Mon, Jan 4, 2016 at 10:13 PM, Yonik Seeley  wrote:
>>
>> On Mon, Jan 4, 2016 at 11:35 AM, Jack Krupansky
>>  wrote:
>> > Probably back-compat for the Solr plugin API as well - people have
>> > custom
>> > plugins that should work without a needed refactor. That would include
>> > custom tokenizers, token filters, char filters, query parsers, request
>> > handlers.
>> >
>> > That probably implies back-compat for the major APIs for Solr core as
>> > well -
>> > the code that plugins need to call.
>>
>> That reflects our current policy (i.e. things like
>> SolrIndexSearcher/QueryCommand/QueryResult would be covered).
>>
>> I think Anshum is looking for something that would allow the refactors
>> in this issue to be committed to 5.5
>> https://issues.apache.org/jira/browse/SOLR-8475
>>
>> -Yonik
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
>
> --
> Anshum Gupta



-- 
-
Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Breaking Java back-compat in Solr

[Alexandre Rafalovitch]

Solr plugin story is muddy enough as it is. Plugins are hard to find,
share. So, in my eyes, breaking them is not a big effect as if we had
a big active registry.

I guess this is opening the brackets, not the original issue. Feel
free to treat that as an invitation for a separate discussion, in an
its own thread at any point.

Regards,
   Alex.

Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 5 January 2016 at 00:03, Noble Paul  wrote:
> The fact that users can write plugins and the plugins can access
> almost any public methods make it extremely hard to remain backward
> compatible. However , we should not hesitate to break backcompat (if
> required). Users have the choice of sticking to the old version and
> rewrite his plugins using the nw API.
> However public/REST APIs should not break backcompat in a minor verion
>
> On Mon, Jan 4, 2016 at 10:17 PM, Anshum Gupta  wrote:
>> Yes, I'm looking at a long term consensus on issues like SOLR-8475.
>>
>> It would make it possible to improve the code without having to wait until a
>> major version is released.
>>
>> On Mon, Jan 4, 2016 at 10:13 PM, Yonik Seeley  wrote:
>>>
>>> On Mon, Jan 4, 2016 at 11:35 AM, Jack Krupansky
>>>  wrote:
>>> > Probably back-compat for the Solr plugin API as well - people have
>>> > custom
>>> > plugins that should work without a needed refactor. That would include
>>> > custom tokenizers, token filters, char filters, query parsers, request
>>> > handlers.
>>> >
>>> > That probably implies back-compat for the major APIs for Solr core as
>>> > well -
>>> > the code that plugins need to call.
>>>
>>> That reflects our current policy (i.e. things like
>>> SolrIndexSearcher/QueryCommand/QueryResult would be covered).
>>>
>>> I think Anshum is looking for something that would allow the refactors
>>> in this issue to be committed to 5.5
>>> https://issues.apache.org/jira/browse/SOLR-8475
>>>
>>> -Yonik
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>
>>
>>
>> --
>> Anshum Gupta
>
>
>
> --
> -
> Noble Paul
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org