Re: Next Solr Release - 5.5.1 or 6.0 ?

[Shawn Heisey]

On 3/25/2016 5:32 PM, Shawn Heisey wrote:
> You are correct.  Solr (Lucene, actually) only maintains compatibility
> with indexes from the previous major version, so 6.x will read 5.x
> indexes, but not indexes built by 4.x or earlier.

Further clarification on this specific point.  This is your TL;DR
warning.  It might be more info than you wanted:

This compatibility assumes that the underlying data type is still
present in the new version.  As an example of where this becomes
important:  Solr's IntField class was deprecated in version 4.8.  It had
not been prominently featured in Solr example schemas for a VERY long
time, though.  The "int" type in examples has been TrieIntField as far
back as 3.x versions, and possibly longer.

If you have an index containing fields built by a 4.x fieldType using
IntField, you cannot use that index in 5.x, even though the index
*format* is readable by 5.x.  The code to access an IntField value from
the index was removed from 5.x in accordance with project policy on
deprecated code.  If you try changing the type in 5.x to TrieIntField,
the recommended replacement, that won't work either, because
TrieIntField encodes the data on the disk differently than IntField did,
and cannot correctly read the data originally written by IntField in the
4.x version.

Thanks,
Shawn

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Shawn Heisey]

On 3/24/2016 9:45 AM, Jack Krupansky wrote:
> Does anybody know if we have doc on the recommended process for upgrading
> data after upgrading Solr? Sure the upgraded version will work fine with
> that old data, but unless the data is upgraded, the user can't then upgrade
> to the next major release after that. This is a case in point - the user is
> on 4.x and upgrades to 5.x with that 4.x data, but will want to upgrade to
> 6.x shortly, but that will require the 4.x data to be rewritten
> (force-merged?) to 5.x first.

In response to Bram's earlier message:

You are correct.  Solr (Lucene, actually) only maintains compatibility
with indexes from the previous major version, so 6.x will read 5.x
indexes, but not indexes built by 4.x or earlier.

Back to this message:

Running an optimize (force merge) should convert the index to the new
format.  There is also an offline index converter that I've never used,
which is probably the preferred method.

I take a slightly different approach -- when I upgrade (from *any*
version), I delete all the data directories, let Solr recreate them on
restart, and rebuild the indexes from the original data source.  I do
this to avoid *any* potential problems related to upgrading.

Thanks,
Shawn

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Jack Krupansky]

Thanks, Erick, I had forgotten about that. I did find one short reference
to it in the doc: "Be sure to run the Lucene IndexUpgrader included with
Solr 4.10 if you might still have old 3x formatted segments in your index.
Alternatively: fully optimize your index with Solr 4.10 to make sure it
consists only of one up-to-date index segment."

See:
https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5

Note to doc guys and committers: That section needs to be replaced with
"Major Changes form Solr 5 to Solr 6".

Also, that IU reference doesn't link to any doc, even the Lucene Javadoc:
https://lucene.apache.org/core/5_5_0/core/org/apache/lucene/index/IndexUpgrader.html

Feels like there should be some Solr doc as well. For example, can Solr be
running, or does it (each node if SolrCloud) need to be shut down first.
And note that it's needed for each collection. Presumably the collections
can be upgraded in parallel since they are distinct directories. It would
be nice to have a SolrIndexUpgrader to run in one shot and discover and
upgrade all Solr collections.

-- Jack Krupansky

On Thu, Mar 24, 2016 at 12:16 PM, Erick Erickson 
wrote:

> There's always the IndexUpgrader, one could run the 5x version against
> a 4x index and have a 5x-compatible index that would then be readable
> by 6x OOB.
>
> A bit convoluted to be sure.
>
> Erick
>
> On Thu, Mar 24, 2016 at 8:49 AM, Yonik Seeley  wrote:
> > On Thu, Mar 24, 2016 at 11:45 AM, Yonik Seeley 
> wrote:
> >>> I've been led to understand that 6.X (at least the Lucene part?) won't
> >>> be backwards compatible with 4.X data. 5.5 at least works fine with
> data
> >>> files from 4.7, for instance.
> >
> > It really doesn't seem like much changed at the lucene index-format
> > level from 5 to 6...
> > it makes one wonder how much work would be involved in allowing Lucene
> > 6 to directly read a newer 4.x index... maybe it's just down to
> > version strings in the index and not much else?
> >
> > -Yonik
>

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yonik Seeley]

On Thu, Mar 24, 2016 at 12:32 PM, Tomás Fernández Löbbe
 wrote:
>>
>>
>> Not to mention the fact that Solr 6 is using deprecated Lucene 6
>> numeric types if those are removed in Lucene 7, then what?
>>
>> I believe this is going to be an issue. We have SOLR-8396
>  open, but it doesn't look
> like it's going to make it to 6.0 (I tried to look at it but I didn't have
> time the past weeks). We'll have to support it until Solr 8 I guess.

Even if it did make it for 6.0, it seems like someone couldn't upgrade
from 6->7 (future) without reindexing?
I don't think the IndexUpgrade tool is going to migrate from
(Trie)NumericField->PointField, right?

-Yonik

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yago Riveiro]

I did the IndexUpgrade path to upgrade my 4.x index to 5.x (15 terabytes of
data an growing), It wasn't an easy task to do it without downtime,
IndexUpgrade doesn't work if the replica is loaded.  
  
With 12T of data re-index is like a no-no operation (the time expended to do
the re-index can take several months).

  

Optimize one replica at a time doesn't work (All replicas are optimize at the
same time) killing CPU an IO and as result the cluster.

  

Conclusion, if I need to do it again to upgrade to a newer version of Solr I'm
in literally in troubles ...  
  

\--

  

/Yago Riveiro

> On Mar 24 2016, at 4:32 pm, Tomás Fernández Löbbe
 wrote:  

>

> >  
>  
> Not to mention the fact that Solr 6 is using deprecated Lucene 6  
> numeric types if those are removed in Lucene 7, then what?  
>  
> I believe this is going to be an issue. We have SOLR-8396  
; open, but it doesn't
look  
like it's going to make it to 6.0 (I tried to look at it but I didn't have  
time the past weeks). We'll have to support it until Solr 8 I guess.

>

> Tomás

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Tomás Fernández Löbbe]

>
>
> Not to mention the fact that Solr 6 is using deprecated Lucene 6
> numeric types if those are removed in Lucene 7, then what?
>
> I believe this is going to be an issue. We have SOLR-8396
 open, but it doesn't look
like it's going to make it to 6.0 (I tried to look at it but I didn't have
time the past weeks). We'll have to support it until Solr 8 I guess.

Tomás

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yonik Seeley]

On Thu, Mar 24, 2016 at 12:16 PM, Erick Erickson
 wrote:
> There's always the IndexUpgrader, one could run the 5x version against
> a 4x index and have a 5x-compatible index that would then be readable
> by 6x OOB.

This may be the last time that will work.  See the thread on the
changes to all the numeric types... it doesn't seem like the
IndexUpgrader will be able to handle that transition?

Not to mention the fact that Solr 6 is using deprecated Lucene 6
numeric types if those are removed in Lucene 7, then what?

-Yonik

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Erick Erickson]

There's always the IndexUpgrader, one could run the 5x version against
a 4x index and have a 5x-compatible index that would then be readable
by 6x OOB.

A bit convoluted to be sure.

Erick

On Thu, Mar 24, 2016 at 8:49 AM, Yonik Seeley  wrote:
> On Thu, Mar 24, 2016 at 11:45 AM, Yonik Seeley  wrote:
>>> I've been led to understand that 6.X (at least the Lucene part?) won't
>>> be backwards compatible with 4.X data. 5.5 at least works fine with data
>>> files from 4.7, for instance.
>
> It really doesn't seem like much changed at the lucene index-format
> level from 5 to 6...
> it makes one wonder how much work would be involved in allowing Lucene
> 6 to directly read a newer 4.x index... maybe it's just down to
> version strings in the index and not much else?
>
> -Yonik

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yonik Seeley]

On Thu, Mar 24, 2016 at 11:38 AM, Bram Van Dam  wrote:
> On 23/03/16 15:50, Yonik Seeley wrote:
>> Kind of a unique situation for a dot-oh release, but from the Solr
>> perspective, 6.0 should have *fewer* bugs than 5.5 (for those features
>> in 5.5 at least)... we've been squashing a bunch of docValue related
>> issues.
>
> I've been led to understand that 6.X (at least the Lucene part?) won't
> be backwards compatible with 4.X data. 5.5 at least works fine with data
> files from 4.7, for instance. With that in mind, at least from my
> selfish perspective, applying fixes to 5.X would be much appreciated ;-)

I hear ya...
In the event that someone volunteers to make a 5.6 or a 5.5.1 release,
we should have a big back-porting party :-)

-Yonik

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yonik Seeley]

On Thu, Mar 24, 2016 at 11:45 AM, Yonik Seeley  wrote:
>> I've been led to understand that 6.X (at least the Lucene part?) won't
>> be backwards compatible with 4.X data. 5.5 at least works fine with data
>> files from 4.7, for instance.

It really doesn't seem like much changed at the lucene index-format
level from 5 to 6...
it makes one wonder how much work would be involved in allowing Lucene
6 to directly read a newer 4.x index... maybe it's just down to
version strings in the index and not much else?

-Yonik

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Jack Krupansky]

Does anybody know if we have doc on the recommended process for upgrading
data after upgrading Solr? Sure the upgraded version will work fine with
that old data, but unless the data is upgraded, the user can't then upgrade
to the next major release after that. This is a case in point - the user is
on 4.x and upgrades to 5.x with that 4.x data, but will want to upgrade to
6.x shortly, but that will require the 4.x data to be rewritten
(force-merged?) to 5.x first.

-- Jack Krupansky

On Thu, Mar 24, 2016 at 11:38 AM, Bram Van Dam  wrote:

> On 23/03/16 15:50, Yonik Seeley wrote:
> > Kind of a unique situation for a dot-oh release, but from the Solr
> > perspective, 6.0 should have *fewer* bugs than 5.5 (for those features
> > in 5.5 at least)... we've been squashing a bunch of docValue related
> > issues.
>
> I've been led to understand that 6.X (at least the Lucene part?) won't
> be backwards compatible with 4.X data. 5.5 at least works fine with data
> files from 4.7, for instance. With that in mind, at least from my
> selfish perspective, applying fixes to 5.X would be much appreciated ;-)
>
>  - Bram
>
>
>

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Bram Van Dam]

On 23/03/16 15:50, Yonik Seeley wrote:
> Kind of a unique situation for a dot-oh release, but from the Solr
> perspective, 6.0 should have *fewer* bugs than 5.5 (for those features
> in 5.5 at least)... we've been squashing a bunch of docValue related
> issues.

I've been led to understand that 6.X (at least the Lucene part?) won't
be backwards compatible with 4.X data. 5.5 at least works fine with data
files from 4.7, for instance. With that in mind, at least from my
selfish perspective, applying fixes to 5.X would be much appreciated ;-)

 - Bram

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Yonik Seeley]

Kind of a unique situation for a dot-oh release, but from the Solr
perspective, 6.0 should have *fewer* bugs than 5.5 (for those features
in 5.5 at least)... we've been squashing a bunch of docValue related
issues.
-Yonik

On Wed, Mar 23, 2016 at 5:19 AM, Alessandro Benedetti
 wrote:
> Thanks Erick,
> I thought the same, better to wait a little bit for a 6.x migration.
> Let give it some time to be released and pissibly bug fixed :)
> I will follow your suggestion, migrate from 4.10.2 to 5.5.0, a possible
> 5.5.1 should be trivial to upgrade later.
>
> Cheers
>
> On Tue, Mar 22, 2016 at 4:45 PM, Erick Erickson 
> wrote:
>
>> No real plans have been made that I know of for a 5.5.1 release. What
>> happens is that 5.5 was cut as, potentially, the last 5x release. Some
>> small fixes are still back-ported "just in case" there's a 5.5.1
>> release.
>>
>> As for 6.0, that's something of a moving target currently, we're still
>> flushing out issues that need to be fixed first. IOW, there's no good
>> way to predict when it would be, the 4 weeks is just an estimate.
>> Besides, I always like to be a little cautious about using a X.0
>> release.
>>
>> the "bottom line" here is that I'd go with 5.5 for the time being. If
>> a 5.5.1 comes out it should be a really simple upgrade to do if the
>> fixes are important to you.
>>
>> Best,
>> Erick
>>
>> On Tue, Mar 22, 2016 at 8:51 AM, Alessandro Benedetti
>>  wrote:
>> > Hi gents,
>> > I am planning a version upgrade, is it possible to know the next upcoming
>> > version ?
>> > From the Solr news I see the next one will be  Solr 6.0 in 4 weeks
>> > approximately.
>> >
>> > But from Jira I see also the 5.5.1 with 8 Jira  issues in it.
>> > Is it possible to have an estimation of the release dates ?
>> > Is 5.5.1 coming out soon ?
>> >
>> > Cheers
>> >
>> > --
>> > --
>> >
>> > Benedetti Alessandro
>> > Visiting card : http://about.me/alessandro_benedetti
>> >
>> > "Tyger, tyger burning bright
>> > In the forests of the night,
>> > What immortal hand or eye
>> > Could frame thy fearful symmetry?"
>> >
>> > William Blake - Songs of Experience -1794 England
>>
>
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Alessandro Benedetti]

Thanks Erick,
I thought the same, better to wait a little bit for a 6.x migration.
Let give it some time to be released and pissibly bug fixed :)
I will follow your suggestion, migrate from 4.10.2 to 5.5.0, a possible
5.5.1 should be trivial to upgrade later.

Cheers

On Tue, Mar 22, 2016 at 4:45 PM, Erick Erickson 
wrote:

> No real plans have been made that I know of for a 5.5.1 release. What
> happens is that 5.5 was cut as, potentially, the last 5x release. Some
> small fixes are still back-ported "just in case" there's a 5.5.1
> release.
>
> As for 6.0, that's something of a moving target currently, we're still
> flushing out issues that need to be fixed first. IOW, there's no good
> way to predict when it would be, the 4 weeks is just an estimate.
> Besides, I always like to be a little cautious about using a X.0
> release.
>
> the "bottom line" here is that I'd go with 5.5 for the time being. If
> a 5.5.1 comes out it should be a really simple upgrade to do if the
> fixes are important to you.
>
> Best,
> Erick
>
> On Tue, Mar 22, 2016 at 8:51 AM, Alessandro Benedetti
>  wrote:
> > Hi gents,
> > I am planning a version upgrade, is it possible to know the next upcoming
> > version ?
> > From the Solr news I see the next one will be  Solr 6.0 in 4 weeks
> > approximately.
> >
> > But from Jira I see also the 5.5.1 with 8 Jira  issues in it.
> > Is it possible to have an estimation of the release dates ?
> > Is 5.5.1 coming out soon ?
> >
> > Cheers
> >
> > --
> > --
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Next Solr Release - 5.5.1 or 6.0 ?

[Erick Erickson]

No real plans have been made that I know of for a 5.5.1 release. What
happens is that 5.5 was cut as, potentially, the last 5x release. Some
small fixes are still back-ported "just in case" there's a 5.5.1
release.

As for 6.0, that's something of a moving target currently, we're still
flushing out issues that need to be fixed first. IOW, there's no good
way to predict when it would be, the 4 weeks is just an estimate.
Besides, I always like to be a little cautious about using a X.0
release.

the "bottom line" here is that I'd go with 5.5 for the time being. If
a 5.5.1 comes out it should be a really simple upgrade to do if the
fixes are important to you.

Best,
Erick

On Tue, Mar 22, 2016 at 8:51 AM, Alessandro Benedetti
 wrote:
> Hi gents,
> I am planning a version upgrade, is it possible to know the next upcoming
> version ?
> From the Solr news I see the next one will be  Solr 6.0 in 4 weeks
> approximately.
>
> But from Jira I see also the 5.5.1 with 8 Jira  issues in it.
> Is it possible to have an estimation of the release dates ?
> Is 5.5.1 coming out soon ?
>
> Cheers
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England

Next Solr Release - 5.5.1 or 6.0 ?

[Alessandro Benedetti]

Hi gents,
I am planning a version upgrade, is it possible to know the next upcoming
version ?
>From the Solr news I see the next one will be  Solr 6.0 in 4 weeks
approximately.

But from Jira I see also the 5.5.1 with 8 Jira  issues in it.
Is it possible to have an estimation of the release dates ?
Is 5.5.1 coming out soon ?

Cheers

-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England