Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Paras Lehana]

Hi Ken,

I also recommend at least reading if not following "Taking Solr to
Production":
https://lucene.apache.org/solr/guide/8_4/taking-solr-to-production.html.

Following this cleared my doubts regarding upgradation and core referencing
while made upgradation very easy and fast.

While starting Solr, you can also define Solr Home (where your older core
lives) by using -s option.



On Wed, 25 Dec 2019 at 21:44, David Hastings  wrote:

> Exactly. Although I’m a bit curious why your going a .1 version up, I
> always wait until an x2, so I won’t be upgrading until 9.3
>
> > On Dec 25, 2019, at 9:45 AM, Erick Erickson 
> wrote:
> >
> > Should work. At any rate, just try it. Since all you’re doing is
> copying data, even if the new installation doesn’t work you still have the
> original.
> >
> >> On Dec 25, 2019, at 1:35 AM, Ken Walker  wrote:
> >>
> >> Hello Erick,
> >>
> >> Thanks for your reply!
> >>
> >> You mean that, we should follow below steps right?
> >> Here is the data directory path :
> >> solr/solr-8.2.0/server/solr/product/item_core/data
> >>
> >> STEPS :-
> >> 1. Stop old solr-8.2.0 server
> >> 2. Copy data directory (from old solr version to new solr version)
> >> copy solr/solr-8.2.1/server/solr/product/item_core/data to
> >> solr/solr-8.3.1/server/solr/product/item_core/data
> >> 3. Start new solr version solr-8.3.1
> >>
> >> Is it correct way to copy just index only from old to new solr version?
> >> Is it lose any data or anything break in new solr version ?
> >>
> >> Thanks in advance!
> >> -Ken
> >
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
*
*

 

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[David Hastings]

Exactly. Although I’m a bit curious why your going a .1 version up, I always 
wait until an x2, so I won’t be upgrading until 9.3 

> On Dec 25, 2019, at 9:45 AM, Erick Erickson  wrote:
> 
> Should work. At any rate, just try it. Since all you’re doing is copying 
> data, even if the new installation doesn’t work you still have the original.
> 
>> On Dec 25, 2019, at 1:35 AM, Ken Walker  wrote:
>> 
>> Hello Erick,
>> 
>> Thanks for your reply!
>> 
>> You mean that, we should follow below steps right?
>> Here is the data directory path :
>> solr/solr-8.2.0/server/solr/product/item_core/data
>> 
>> STEPS :-
>> 1. Stop old solr-8.2.0 server
>> 2. Copy data directory (from old solr version to new solr version)
>> copy solr/solr-8.2.1/server/solr/product/item_core/data to
>> solr/solr-8.3.1/server/solr/product/item_core/data
>> 3. Start new solr version solr-8.3.1
>> 
>> Is it correct way to copy just index only from old to new solr version?
>> Is it lose any data or anything break in new solr version ?
>> 
>> Thanks in advance!
>> -Ken
> 

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Erick Erickson]

Should work. At any rate, just try it. Since all you’re doing is copying data, 
even if the new installation doesn’t work you still have the original.

> On Dec 25, 2019, at 1:35 AM, Ken Walker  wrote:
> 
> Hello Erick,
> 
> Thanks for your reply!
> 
> You mean that, we should follow below steps right?
> Here is the data directory path :
> solr/solr-8.2.0/server/solr/product/item_core/data
> 
> STEPS :-
> 1. Stop old solr-8.2.0 server
> 2. Copy data directory (from old solr version to new solr version)
> copy solr/solr-8.2.1/server/solr/product/item_core/data to
> solr/solr-8.3.1/server/solr/product/item_core/data
> 3. Start new solr version solr-8.3.1
> 
> Is it correct way to copy just index only from old to new solr version?
> Is it lose any data or anything break in new solr version ?
> 
> Thanks in advance!
> -Ken

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Ken Walker]

Hello Erick,

Thanks for your reply!

You mean that, we should follow below steps right?
Here is the data directory path :
solr/solr-8.2.0/server/solr/product/item_core/data

STEPS :-
1. Stop old solr-8.2.0 server
2. Copy data directory (from old solr version to new solr version)
copy solr/solr-8.2.1/server/solr/product/item_core/data to
solr/solr-8.3.1/server/solr/product/item_core/data
3. Start new solr version solr-8.3.1

Is it correct way to copy just index only from old to new solr version?
Is it lose any data or anything break in new solr version ?

Thanks in advance!
-Ken

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Erick Erickson]

Here’s the very simplest way:
1> shut down your 8.2 Solr instance
2> install your 8.3.1 instance on the same machine
3> when you start your 8.3.1 instance, specify the environment variable 
SOLR_HOME to point to the same one you used in 8.2

If you don’t know what SOLR_HOME used to point to, bring up your 8.2 instance 
first and look at the admin UI, your environment variables will point there.

NOTE: If yo do it this way, you ma _NOT_ have both 8.2 and 8.3.1 running the 
same time.

Best,
Erick

> On Dec 24, 2019, at 5:54 AM, Ken Walker  wrote:
> 
> Hello Jörn,
> 
> Thanks for your reply!
> 
> As per Shawn "Why not just copy the index and use it directly rather
> than importing it?  Solr 8.x can directly use indexes built by
> versions back to 7.0.0." in previous mail comment.
> 
> Is it possible and how we can do that ?
> 
> Thanks in advance
> - Ken
> 
> On Tue, Dec 24, 2019 at 3:26 PM Jörn Franke  wrote:
>> 
>> It seems that you got this handed over with little documentation. You have 
>> to explore what the import handler does. This is a custom configuration that 
>> you need to check how it works.
>> 
>> Then as already said. You can simply install another version of Solr if you 
>> are within a Solr major version 8.x in Linux is simply a symbolic link 
>> pointing from one Solr version to the other. In this way you can easily 
>> switch back as well.
>> 
>> Finally, check your memory consumption. Normally heap is significant smaller 
>> then the total available memory as the non-heap memory is used by Solr for 
>> caching.
>> 
>> If you have 8g mb of heap I would expect that the total amount of memory 
>> available is more than 32 gb.
>> As always it depends, but maybe you can give more details on no of cores, 
>> heap memory, total memory and if other processes than Solr run on the 
>> machine.
>> 
>>> Am 24.12.2019 um 05:59 schrieb Ken Walker :
>>> 
>>> Hello,
>>> 
>>> We are using solr version 8.2.0 in our production server.
>>> 
>>> We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
>>> version but we have faced out of memory error while importing data and
>>> then we have extended memory in our server and then again start
>>> importing process but it has work too slowy for 8GB data ( it has
>>> taken more than 2 days for importing data from solr 8.2.0 version to
>>> solr 8.3.1 version).
>>> 
>>> Could you please help me how we can do it fast for importing 8GB data
>>> from old solr version to new solr version?
>>> 
>>> We are using below command for importing data from one solr version to
>>> another solr version
>>> $ curl 
>>> 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
>>> 
>>> Thanks in advance!
>>> - Ken

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Ken Walker]

Hello Jörn,

Thanks for your reply!

As per Shawn "Why not just copy the index and use it directly rather
than importing it?  Solr 8.x can directly use indexes built by
versions back to 7.0.0." in previous mail comment.

Is it possible and how we can do that ?

Thanks in advance
- Ken

On Tue, Dec 24, 2019 at 3:26 PM Jörn Franke  wrote:
>
> It seems that you got this handed over with little documentation. You have to 
> explore what the import handler does. This is a custom configuration that you 
> need to check how it works.
>
> Then as already said. You can simply install another version of Solr if you 
> are within a Solr major version 8.x in Linux is simply a symbolic link 
> pointing from one Solr version to the other. In this way you can easily 
> switch back as well.
>
> Finally, check your memory consumption. Normally heap is significant smaller 
> then the total available memory as the non-heap memory is used by Solr for 
> caching.
>
> If you have 8g mb of heap I would expect that the total amount of memory 
> available is more than 32 gb.
> As always it depends, but maybe you can give more details on no of cores, 
> heap memory, total memory and if other processes than Solr run on the machine.
>
> > Am 24.12.2019 um 05:59 schrieb Ken Walker :
> >
> > Hello,
> >
> > We are using solr version 8.2.0 in our production server.
> >
> > We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> > version but we have faced out of memory error while importing data and
> > then we have extended memory in our server and then again start
> > importing process but it has work too slowy for 8GB data ( it has
> > taken more than 2 days for importing data from solr 8.2.0 version to
> > solr 8.3.1 version).
> >
> > Could you please help me how we can do it fast for importing 8GB data
> > from old solr version to new solr version?
> >
> > We are using below command for importing data from one solr version to
> > another solr version
> > $ curl 
> > 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
> >
> > Thanks in advance!
> > - Ken

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Jörn Franke]

It seems that you got this handed over with little documentation. You have to 
explore what the import handler does. This is a custom configuration that you 
need to check how it works.

Then as already said. You can simply install another version of Solr if you are 
within a Solr major version 8.x in Linux is simply a symbolic link pointing 
from one Solr version to the other. In this way you can easily switch back as 
well.

Finally, check your memory consumption. Normally heap is significant smaller 
then the total available memory as the non-heap memory is used by Solr for 
caching.

If you have 8g mb of heap I would expect that the total amount of memory 
available is more than 32 gb.
As always it depends, but maybe you can give more details on no of cores, heap 
memory, total memory and if other processes than Solr run on the machine.

> Am 24.12.2019 um 05:59 schrieb Ken Walker :
> 
> Hello,
> 
> We are using solr version 8.2.0 in our production server.
> 
> We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> version but we have faced out of memory error while importing data and
> then we have extended memory in our server and then again start
> importing process but it has work too slowy for 8GB data ( it has
> taken more than 2 days for importing data from solr 8.2.0 version to
> solr 8.3.1 version).
> 
> Could you please help me how we can do it fast for importing 8GB data
> from old solr version to new solr version?
> 
> We are using below command for importing data from one solr version to
> another solr version
> $ curl 
> 'http://IP-ADDRESS:8983/solr/items/dataimport?command=full-import&clean=true&commit=false&wt=json&indent=true&verbose=false&optimize=false&debug=false'
> 
> Thanks in advance!
> - Ken

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Ken Walker]

Hello Shawn,

Thanks for your reply!

Actually we don't know how its works ( just copy the index ) so could
you please give us some reference urls or any steps for it?

Thanks in advance
- Ken

On Tue, Dec 24, 2019 at 11:56 AM Shawn Heisey  wrote:
>
> On 12/23/2019 9:58 PM, Ken Walker wrote:
> > We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
> > version but we have faced out of memory error while importing data and
> > then we have extended memory in our server and then again start
> > importing process but it has work too slowy for 8GB data ( it has
> > taken more than 2 days for importing data from solr 8.2.0 version to
> > solr 8.3.1 version).
> >
> > Could you please help me how we can do it fast for importing 8GB data
> > from old solr version to new solr version?
>
> Why not just copy the index and use it directly rather than importing
> it?  Solr 8.x can directly use indexes built by versions back to 7.0.0.
>
> Thanks,
> Shawn

Re: Help for importing large data (approx. 8GB) from old solr version to new solr version

[Shawn Heisey]

On 12/23/2019 9:58 PM, Ken Walker wrote:

We are upgrading solr version from solr 8.2.0 version to solr 8.3.1
version but we have faced out of memory error while importing data and
then we have extended memory in our server and then again start
importing process but it has work too slowy for 8GB data ( it has
taken more than 2 days for importing data from solr 8.2.0 version to
solr 8.3.1 version).

Could you please help me how we can do it fast for importing 8GB data
from old solr version to new solr version?


Why not just copy the index and use it directly rather than importing 
it?  Solr 8.x can directly use indexes built by versions back to 7.0.0.


Thanks,
Shawn