Reindex after schema change options

[tedsolr]

Not all my fields use docValues. This is going to be a problem in the future.
Once I change the schema.xml to use docValues for these certain field types,
how do I reindex the data in place - without starting from the source?

I'm aware of lucene's IndexUpgrader but that will only ensure a correct
lucene match version. I'm not changing that. Could I use SolrJ to walk
through the documents and "touch" each one and do an atomic update on the
fields that have changed? (all the fields I care about are stored)

Thanks, Ted
v5.2.1



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Reindex-after-schema-change-options-tp4303395.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merge policy

[Walter Underwood]

That distribution of segment sizes seems odd. Why so many medium-large segments?

Are there custom settings for merge policy? I think the default policy would 
avoid so many segments that are mostly deleted documents.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Oct 27, 2016, at 9:40 AM, Shawn Heisey  wrote:
> 
> On 10/27/2016 9:50 AM, Yonik Seeley wrote:
>> On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson 
>> wrote:
>>> Thanks for the answer! Do you know if there is a way to trigger an
>>> optimize for only 1 shard and not the whole collection at once? 
>> Adding a "distrib=false" parameter should work I think. 
> 
> Last time I checked, which I admit has been a little while, optimize
> ignored distrib and proceeded with a sequential optimize of every core
> in the collection.
> 
> Thanks,
> Shawn
> 

Re: Merge policy

[Shawn Heisey]

On 10/27/2016 9:50 AM, Yonik Seeley wrote:
> On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson 
> wrote:
>> Thanks for the answer! Do you know if there is a way to trigger an
>> optimize for only 1 shard and not the whole collection at once? 
> Adding a "distrib=false" parameter should work I think. 

Last time I checked, which I admit has been a little while, optimize
ignored distrib and proceeded with a sequential optimize of every core
in the collection.

Thanks,
Shawn

Re: Solr Cloud A/B Deployment Issue

[jimtronic]

Great. Thanks for the work on this patch!

Jim



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Cloud-A-B-Deployment-Issue-tp4302810p4303357.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Comparing between 2 String fields

[Zheng Lin Edwin Yeo]

Hi Emir,

Thanks for your reply.

Does all these functions only works for Solr 6.x onwards?

Regards,
Edwin

On 27 October 2016 at 20:13, Emir Arnautovic 
wrote:

> Hi Edwin,
>
> You can use functions to do that, e.g.
>
> fq={!frange l=1}strdist(field1,field2, edit)
>
> Solr now has eq func as well, so you can use that one in case you are
> running latest version.
>
> HTH,
> Emir
>
>
> On 27.10.2016 13:39, Zheng Lin Edwin Yeo wrote:
>
>> Hi,
>>
>> Is it possible to compare between 2 String fields in Solr?
>> Like for example, I want to find those record which field1=field2, and
>> list
>> them out.
>>
>> I'm using Solr 6.2.1
>>
>> Regards,
>> Edwin
>>
>>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>

Re: Merge policy

[Yonik Seeley]

On Thu, Oct 27, 2016 at 9:56 AM, Arkadi Colson  wrote:

> Thanks for the answer!
> Do you know if there is a way to trigger an optimize for only 1 shard and
> not the whole collection at once?
>

Adding a "distrib=false" parameter should work I think.

-Yonik

Re: Merge policy

[Erick Erickson]

Why do you think you need to get rid of the deleted data? During normal
indexing, these will be "merged away". Optimizing has some downsides
for continually changing indexes, in particular since the default
tieredmergepolicy tries to merge "like size" segments, deletions will
accumulate in your one large segment and the percentage of
deleted documents may get even higher.

Unless there's some measurable performance gain that the users
will notice, I'd just leave this alone.

The exception here is if you have, say, an index that changes rarely
in which case optimizing then makes more sense.

Best,
Erick

On Thu, Oct 27, 2016 at 6:56 AM, Arkadi Colson  wrote:

> Thanks for the answer!
> Do you know if there is a way to trigger an optimize for only 1 shard and
> not the whole collection at once?
>
> On 27-10-16 15:30, Pushkar Raste wrote:
>
> Try commit with expungeDeletes="true"
>
> I am not sure if it will merge old segments that have deleted documents.
>
> In the worst case you can 'optimize' your index which should take care of
> removing deleted document
>
> On Oct 27, 2016 4:20 AM, "Arkadi Colson"  wrote:
>
>> Hi
>>
>> As you can see in the screenshot above in the oldest segments there are a
>> lot of deletions. In total the shard has about 26% deletions. How can I get
>> rid of them so the index will be smaller again?
>> Can this only be done with an optimize or does it also depend on the
>> merge policy? If it also depends also on the merge policy which one should
>> I choose then?
>>
>> Thanks!
>>
>> BR,
>> Arkadi
>>
>
>

Re: 'solr zk upconfig' etc not working on windows since 6.1 at least

[xavier jmlucjav]

done, with simple patch https://issues.apache.org/jira/browse/SOLR-9697

On Thu, Oct 27, 2016 at 4:21 PM, xavier jmlucjav  wrote:

> sure, will do, I tried before but I could not create a Jira, now I can,
> not sure what was happening.
>
> On Thu, Oct 27, 2016 at 3:14 PM, Shalin Shekhar Mangar <
> shalinman...@gmail.com> wrote:
>
>> Would you mind opening a jira issue and give a patch (diff)? 6.3 is coming
>> out soon and we'd have to hurry if this fix has to go in.
>>
>> On Thu, Oct 27, 2016 at 6:32 PM, xavier jmlucjav 
>> wrote:
>>
>> > Correcting myself here, I was wrong about the cause (I had already
>> messed
>> > with the script).
>> >
>> > I made it work by commenting out line 1261 (the number might be a bit
>> off
>> > as I have modified the script, but hopefully its easy to see where):
>> >
>> > ) ELSE IF "%1"=="/?" (
>> >   goto zk_usage
>> > ) ELSE IF "%1"=="-h" (
>> >   goto zk_usage
>> > ) ELSE IF "%1"=="-help" (
>> >   goto zk_usage
>> > ) ELSE IF "!ZK_SRC!"=="" (
>> >   if not "%~1"=="" (
>> > goto set_zk_src
>> >   )
>> >  * rem goto zk_usage*
>> > ) ELSE IF "!ZK_DST!"=="" (
>> >   IF "%ZK_OP%"=="cp" (
>> > goto set_zk_dst
>> >   )
>> >   IF "%ZK_OP%"=="mv" (
>> > goto set_zk_dst
>> >   )
>> >   set ZK_DST="_"
>> > ) ELSE IF NOT "%1"=="" (
>> >   set ERROR_MSG="Unrecognized or misplaced zk argument %1%"
>> >
>> > Now upconfig works!
>> >
>> > thanks
>> > xavier
>> >
>> >
>> > On Thu, Oct 27, 2016 at 2:43 PM, xavier jmlucjav 
>> > wrote:
>> >
>> > > hi,
>> > >
>> > > Am I missing something or this is broken in windows? I cannot
>> upconfig,
>> > > the scripts keeps exiting immediately and showing usage, as if I use
>> some
>> > > wrong parameters.  This is on win10, jdk8. But I am pretty sure I saw
>> it
>> > > also on win7 (don't have that around anymore to try)
>> > >
>> > > I think the issue is: there is a SHIFT too much in line 1276 of
>> solr.cmd:
>> > >
>> > > :set_zk_op
>> > > set ZK_OP=%~1
>> > > SHIFT
>> > > goto parse_zk_args
>> > >
>> > > if this SHIFT is removed, then parse_zk_args works (and it does the
>> shift
>> > > itself). But the upconfig hangs, so still it does not work.
>> > >
>> > > this probably was introduced in a851d5f557aefd76c01ac23da076a1
>> 4dc7576d8e
>> > > by Erick (not sure which one :) ) on July 2nd. Master still has this
>> > issue.
>> > > Would be great if this was fixed in the incoming 6.3...
>> > >
>> > > My cmd scripting is not too strong and I did not go further. I
>> searched
>> > > Jira but found nothing. By the way is it not possible to open tickets
>> in
>> > > Jira anymore?
>> > >
>> > > xavier
>> > >
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>

Re: 'solr zk upconfig' etc not working on windows since 6.1 at least

[xavier jmlucjav]

sure, will do, I tried before but I could not create a Jira, now I can, not
sure what was happening.

On Thu, Oct 27, 2016 at 3:14 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> Would you mind opening a jira issue and give a patch (diff)? 6.3 is coming
> out soon and we'd have to hurry if this fix has to go in.
>
> On Thu, Oct 27, 2016 at 6:32 PM, xavier jmlucjav 
> wrote:
>
> > Correcting myself here, I was wrong about the cause (I had already messed
> > with the script).
> >
> > I made it work by commenting out line 1261 (the number might be a bit off
> > as I have modified the script, but hopefully its easy to see where):
> >
> > ) ELSE IF "%1"=="/?" (
> >   goto zk_usage
> > ) ELSE IF "%1"=="-h" (
> >   goto zk_usage
> > ) ELSE IF "%1"=="-help" (
> >   goto zk_usage
> > ) ELSE IF "!ZK_SRC!"=="" (
> >   if not "%~1"=="" (
> > goto set_zk_src
> >   )
> >  * rem goto zk_usage*
> > ) ELSE IF "!ZK_DST!"=="" (
> >   IF "%ZK_OP%"=="cp" (
> > goto set_zk_dst
> >   )
> >   IF "%ZK_OP%"=="mv" (
> > goto set_zk_dst
> >   )
> >   set ZK_DST="_"
> > ) ELSE IF NOT "%1"=="" (
> >   set ERROR_MSG="Unrecognized or misplaced zk argument %1%"
> >
> > Now upconfig works!
> >
> > thanks
> > xavier
> >
> >
> > On Thu, Oct 27, 2016 at 2:43 PM, xavier jmlucjav 
> > wrote:
> >
> > > hi,
> > >
> > > Am I missing something or this is broken in windows? I cannot upconfig,
> > > the scripts keeps exiting immediately and showing usage, as if I use
> some
> > > wrong parameters.  This is on win10, jdk8. But I am pretty sure I saw
> it
> > > also on win7 (don't have that around anymore to try)
> > >
> > > I think the issue is: there is a SHIFT too much in line 1276 of
> solr.cmd:
> > >
> > > :set_zk_op
> > > set ZK_OP=%~1
> > > SHIFT
> > > goto parse_zk_args
> > >
> > > if this SHIFT is removed, then parse_zk_args works (and it does the
> shift
> > > itself). But the upconfig hangs, so still it does not work.
> > >
> > > this probably was introduced in a851d5f557aefd76c01ac23da076a1
> 4dc7576d8e
> > > by Erick (not sure which one :) ) on July 2nd. Master still has this
> > issue.
> > > Would be great if this was fixed in the incoming 6.3...
> > >
> > > My cmd scripting is not too strong and I did not go further. I searched
> > > Jira but found nothing. By the way is it not possible to open tickets
> in
> > > Jira anymore?
> > >
> > > xavier
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Merge policy

[Arkadi Colson]

Thanks for the answer!
Do you know if there is a way to trigger an optimize for only 1 shard 
and not the whole collection at once?



On 27-10-16 15:30, Pushkar Raste wrote:


Try commit with expungeDeletes="true"

I am not sure if it will merge old segments that have deleted documents.

In the worst case you can 'optimize' your index which should take care 
of removing deleted document



On Oct 27, 2016 4:20 AM, "Arkadi Colson" > wrote:


Hi

As you can see in the screenshot above in the oldest segments
there are a lot of deletions. In total the shard has about 26%
deletions. How can I get rid of them so the index will be smaller
again?
Can this only be done with an optimize or does it also depend on
the merge policy? If it also depends also on the merge policy
which one should I choose then?

Thanks!

BR,
Arkadi

Re: Merge policy

[Pushkar Raste]

Try commit with expungeDeletes="true"

I am not sure if it will merge old segments that have deleted documents.

In the worst case you can 'optimize' your index which should take care of
removing deleted document

On Oct 27, 2016 4:20 AM, "Arkadi Colson"  wrote:

> Hi
>
> As you can see in the screenshot above in the oldest segments there are a
> lot of deletions. In total the shard has about 26% deletions. How can I get
> rid of them so the index will be smaller again?
> Can this only be done with an optimize or does it also depend on the merge
> policy? If it also depends also on the merge policy which one should I
> choose then?
>
> Thanks!
>
> BR,
> Arkadi
>

Re: 'solr zk upconfig' etc not working on windows since 6.1 at least

[Shalin Shekhar Mangar]

Would you mind opening a jira issue and give a patch (diff)? 6.3 is coming
out soon and we'd have to hurry if this fix has to go in.

On Thu, Oct 27, 2016 at 6:32 PM, xavier jmlucjav  wrote:

> Correcting myself here, I was wrong about the cause (I had already messed
> with the script).
>
> I made it work by commenting out line 1261 (the number might be a bit off
> as I have modified the script, but hopefully its easy to see where):
>
> ) ELSE IF "%1"=="/?" (
>   goto zk_usage
> ) ELSE IF "%1"=="-h" (
>   goto zk_usage
> ) ELSE IF "%1"=="-help" (
>   goto zk_usage
> ) ELSE IF "!ZK_SRC!"=="" (
>   if not "%~1"=="" (
> goto set_zk_src
>   )
>  * rem goto zk_usage*
> ) ELSE IF "!ZK_DST!"=="" (
>   IF "%ZK_OP%"=="cp" (
> goto set_zk_dst
>   )
>   IF "%ZK_OP%"=="mv" (
> goto set_zk_dst
>   )
>   set ZK_DST="_"
> ) ELSE IF NOT "%1"=="" (
>   set ERROR_MSG="Unrecognized or misplaced zk argument %1%"
>
> Now upconfig works!
>
> thanks
> xavier
>
>
> On Thu, Oct 27, 2016 at 2:43 PM, xavier jmlucjav 
> wrote:
>
> > hi,
> >
> > Am I missing something or this is broken in windows? I cannot upconfig,
> > the scripts keeps exiting immediately and showing usage, as if I use some
> > wrong parameters.  This is on win10, jdk8. But I am pretty sure I saw it
> > also on win7 (don't have that around anymore to try)
> >
> > I think the issue is: there is a SHIFT too much in line 1276 of solr.cmd:
> >
> > :set_zk_op
> > set ZK_OP=%~1
> > SHIFT
> > goto parse_zk_args
> >
> > if this SHIFT is removed, then parse_zk_args works (and it does the shift
> > itself). But the upconfig hangs, so still it does not work.
> >
> > this probably was introduced in a851d5f557aefd76c01ac23da076a14dc7576d8e
> > by Erick (not sure which one :) ) on July 2nd. Master still has this
> issue.
> > Would be great if this was fixed in the incoming 6.3...
> >
> > My cmd scripting is not too strong and I did not go further. I searched
> > Jira but found nothing. By the way is it not possible to open tickets in
> > Jira anymore?
> >
> > xavier
> >
>



-- 
Regards,
Shalin Shekhar Mangar.

Re: 'solr zk upconfig' etc not working on windows since 6.1 at least

[xavier jmlucjav]

Correcting myself here, I was wrong about the cause (I had already messed
with the script).

I made it work by commenting out line 1261 (the number might be a bit off
as I have modified the script, but hopefully its easy to see where):

) ELSE IF "%1"=="/?" (
  goto zk_usage
) ELSE IF "%1"=="-h" (
  goto zk_usage
) ELSE IF "%1"=="-help" (
  goto zk_usage
) ELSE IF "!ZK_SRC!"=="" (
  if not "%~1"=="" (
goto set_zk_src
  )
 * rem goto zk_usage*
) ELSE IF "!ZK_DST!"=="" (
  IF "%ZK_OP%"=="cp" (
goto set_zk_dst
  )
  IF "%ZK_OP%"=="mv" (
goto set_zk_dst
  )
  set ZK_DST="_"
) ELSE IF NOT "%1"=="" (
  set ERROR_MSG="Unrecognized or misplaced zk argument %1%"

Now upconfig works!

thanks
xavier


On Thu, Oct 27, 2016 at 2:43 PM, xavier jmlucjav  wrote:

> hi,
>
> Am I missing something or this is broken in windows? I cannot upconfig,
> the scripts keeps exiting immediately and showing usage, as if I use some
> wrong parameters.  This is on win10, jdk8. But I am pretty sure I saw it
> also on win7 (don't have that around anymore to try)
>
> I think the issue is: there is a SHIFT too much in line 1276 of solr.cmd:
>
> :set_zk_op
> set ZK_OP=%~1
> SHIFT
> goto parse_zk_args
>
> if this SHIFT is removed, then parse_zk_args works (and it does the shift
> itself). But the upconfig hangs, so still it does not work.
>
> this probably was introduced in a851d5f557aefd76c01ac23da076a14dc7576d8e
> by Erick (not sure which one :) ) on July 2nd. Master still has this issue.
> Would be great if this was fixed in the incoming 6.3...
>
> My cmd scripting is not too strong and I did not go further. I searched
> Jira but found nothing. By the way is it not possible to open tickets in
> Jira anymore?
>
> xavier
>

'solr zk upconfig' etc not working on windows since 6.1 at least

[xavier jmlucjav]

hi,

Am I missing something or this is broken in windows? I cannot upconfig, the
scripts keeps exiting immediately and showing usage, as if I use some wrong
parameters.  This is on win10, jdk8. But I am pretty sure I saw it also on
win7 (don't have that around anymore to try)

I think the issue is: there is a SHIFT too much in line 1276 of solr.cmd:

:set_zk_op
set ZK_OP=%~1
SHIFT
goto parse_zk_args

if this SHIFT is removed, then parse_zk_args works (and it does the shift
itself). But the upconfig hangs, so still it does not work.

this probably was introduced in a851d5f557aefd76c01ac23da076a14dc7576d8e by
Erick (not sure which one :) ) on July 2nd. Master still has this issue.
Would be great if this was fixed in the incoming 6.3...

My cmd scripting is not too strong and I did not go further. I searched
Jira but found nothing. By the way is it not possible to open tickets in
Jira anymore?

xavier

Re: Comparing between 2 String fields

[Emir Arnautovic]

Hi Edwin,

You can use functions to do that, e.g.

fq={!frange l=1}strdist(field1,field2, edit)

Solr now has eq func as well, so you can use that one in case you are 
running latest version.


HTH,
Emir

On 27.10.2016 13:39, Zheng Lin Edwin Yeo wrote:

Hi,

Is it possible to compare between 2 String fields in Solr?
Like for example, I want to find those record which field1=field2, and list
them out.

I'm using Solr 6.2.1

Regards,
Edwin



--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/

Comparing between 2 String fields

[Zheng Lin Edwin Yeo]

Hi,

Is it possible to compare between 2 String fields in Solr?
Like for example, I want to find those record which field1=field2, and list
them out.

I'm using Solr 6.2.1

Regards,
Edwin

Merge policy

[Arkadi Colson]

Hi

As you can see in the screenshot above in the oldest segments there are 
a lot of deletions. In total the shard has about 26% deletions. How can 
I get rid of them so the index will be smaller again?
Can this only be done with an optimize or does it also depend on the 
merge policy? If it also depends also on the merge policy which one 
should I choose then?


Thanks!

BR,
Arkadi