firstSearcher and newSearcher loading external files

[Summer Shire]

Hi All,

I use value source parser to load external file that contains popularity
values.



The way I use it is by passing a  "signals" param in the query now I want
to warm up this external cache in solr before the actual traffic starts
coming in.
What is happening is that the cache does not get warmed up in the first or
new searcher
and when the first client query comes in there is a huge delay to load the
cache file. So the first actual query has to take the hit.

I tried to add my function using the following tag in firstSearcher and
newSearcher but that did not work. Any advice ?



Thanks,

Summer

Re: Too many Soft commits and opening searchers realtime

[Summer Shire]

Yonik, Mikhail, Alessandro 

After a lot of digging around and isolation, All u guys were right. I was using 
property based value and there was one place where it was 30 secs and that was 
overriding my main props. 

Also Yonik thanks for the explanation on the real time searcher. I wasn't sure 
if the maxwarmingSearcher error I was getting also had something to do with it. 

Thanks a lot 

> On Jul 8, 2015, at 5:28 AM, Yonik Seeley  wrote:
> 
> A realtime searcher is necessary for internal bookkeeping / uses if a
> normal searcher isn't opened on a commit.
> This searcher doesn't have caches and hence doesn't carry the weight
> that a normal searcher would.  It's also invisible to clients (it
> doesn't change the view of the index for normal searches).
> 
> Your hard autocommit at 8 minutes with openSearcher=false will trigger
> a realtime searcher to open on every 8 minutes along with the hard
> commit.
> 
> -Yonik
> 
> 
>> On Tue, Jul 7, 2015 at 5:29 PM, Summer Shire  wrote:
>> HI All,
>> 
>> Can someone help me understand the following behavior.
>> I have the following maxTimes on hard and soft commits
>> 
>> yet I see a lot of Opening Searchers in the log
>> org.apache.solr.search.SolrIndexSearcher - Opening Searcher@1656a258[main] 
>> realtime
>> also I see a soft commit happening almost every 30 secs
>> org.apache.solr.update.UpdateHandler - start 
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>> 
>> 48
>> false
>> 
>> 
>> 
>> 18
>> 
>> I tried disabling softCommit by setting maxTime to -1.
>> On startup solrCore recognized it and logged "Soft AutoCommit: disabled"
>> but I could still see softCommit=true
>> org.apache.solr.update.UpdateHandler - start 
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>> 
>> -1
>> 
>> 
>> Thanks,
>> Summer

Re: Tlog replay

[Summer Shire]

Thanks Alessandro !

Any idea on why I couldn't curl the solr core and pass the flag param ?


> On Jul 8, 2015, at 7:12 AM, Alessandro Benedetti  
> wrote:
> 
> Hi Summer,
> 
> If you take a look to the CommitUpdateCommand class, you will notice no
> Flag is in there.
> 
> // this is the toString for example
> 
> @Override
> public String toString() {
>  return super.toString() + ",optimize="+optimize
>  +",openSearcher="+openSearcher
>  +",waitSearcher="+waitSearcher
>  +",expungeDeletes="+expungeDeletes
>  +",softCommit="+softCommit
>  +",prepareCommit="+prepareCommit
>  +'}';
> }
> 
> 
> If you then access the UpdateCommand object, you find the flag :
> 
> 
> public static int BUFFERING = 0x0001;// update command is
> being buffered.
> public static int REPLAY= 0x0002;// update command is from
> replaying a log.
> public static int PEER_SYNC= 0x0004; // update command is a
> missing update being provided by a peer.
> public static int IGNORE_AUTOCOMMIT = 0x0008; // this update
> should not count toward triggering of autocommits.
> public static int CLEAR_CACHES = 0x0010; // clear caches
> associated with the update log.  used when applying reordered DBQ
> updates when doing an add.
> 
> So the flag =2 is actually saying that the update command is from
> replaying a log ( which is what you would expect)
> 
> 
> Cheers
> 
> 
> 2015-07-08 3:01 GMT+01:00 Summer Shire :
> 
>> 
>> Hi,
>> 
>> When I restart my solr core the log replay starts and just before it
>> finishes I see the following commit
>> 
>> start
>> commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>> 
>> what does the “flags=2” param do ?
>> 
>> when I try to send that param to the updateHandler manually solr does not
>> like it
>> 
>> curl http://localhost:6600/solr/main/update -H "Content-Type: text/xml"
>> --data-binary '> waitSearcher="false"/>'
>> 
>> 
>> 
>> 400> name="QTime">0Unknown commit
>> parameter 'flags'400
>> 
>> 
>> thanks,
>> Summer
> 
> 
> 
> 
> -- 
> --
> 
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
> 
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
> 
> William Blake - Songs of Experience -1794 England

Tlog replay

[Summer Shire]

Hi,

When I restart my solr core the log replay starts and just before it finishes I 
see the following commit

start 
commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

what does the “flags=2” param do ?

when I try to send that param to the updateHandler manually solr does not like 
it

curl http://localhost:6600/solr/main/update -H "Content-Type: text/xml" 
--data-binary ''



4000Unknown commit 
parameter 'flags'400


thanks,
Summer

Re: Too many Soft commits and opening searchers realtime

[Summer Shire]

No the client lets solr handle it. 


> On Jul 7, 2015, at 2:38 PM, Mike Drob  wrote:
> 
> Are the clients that are posting updates requesting commits?
> 
>> On Tue, Jul 7, 2015 at 4:29 PM, Summer Shire  wrote:
>> 
>> HI All,
>> 
>> Can someone help me understand the following behavior.
>> I have the following maxTimes on hard and soft commits
>> 
>> yet I see a lot of Opening Searchers in the log
>> org.apache.solr.search.SolrIndexSearcher - Opening Searcher@1656a258[main]
>> realtime
>> also I see a soft commit happening almost every 30 secs
>> org.apache.solr.update.UpdateHandler - start
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>> 
>> 48
>> false
>> 
>> 
>> 
>> 18
>> 
>> I tried disabling softCommit by setting maxTime to -1.
>> On startup solrCore recognized it and logged "Soft AutoCommit: disabled"
>> but I could still see softCommit=true
>> org.apache.solr.update.UpdateHandler - start
>> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>> 
>> -1
>> 
>> 
>> Thanks,
>> Summer

Too many Soft commits and opening searchers realtime

[Summer Shire]

HI All,

Can someone help me understand the following behavior.
I have the following maxTimes on hard and soft commits

yet I see a lot of Opening Searchers in the log
org.apache.solr.search.SolrIndexSearcher- Opening Searcher@1656a258[main] 
realtime
also I see a soft commit happening almost every 30 secs
org.apache.solr.update.UpdateHandler - start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}

48
false



18

I tried disabling softCommit by setting maxTime to -1.
On startup solrCore recognized it and logged "Soft AutoCommit: disabled"
but I could still see softCommit=true 
org.apache.solr.update.UpdateHandler - start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}

-1


Thanks,
Summer

Re: optimize status

[Summer Shire]

Upayavira:
I am using solr 4.7 and yes I am using TieredMergePolicy

Erick:
All my boxes have SSD’s and there isn’t a big disparity between qTime and 
response time.
The performance hit on my end is because of the fragmented index files causing 
more disk seeks are you mentioned.
And I tried requesting fewer docs too but that did not help .



> On Jun 30, 2015, at 5:23 AM, Erick Erickson  wrote:
> 
> I've actually seen this happen right in front of my eyes "in the
> field". However, that was a very high-performance environment. My
> assumption was that fragmented index files were causing more disk
> seeks especially for the first-pass query response in distributed
> mode. So, if the problem is similar, it should go away if you test
> requesting fewer docs. Note: This is not a cure for your problem, but
> would be useful for identifying if it's similar to what I saw.
> 
> NOTE: the symptom was a significant disparity between the QTime (which
> does not measure assembling the document) and the response time. _If_
> that's the case and _if_ my theory that disk access is the culprit,
> then SOLR-5478 and SOLR-6810 should be a big help as they remove the
> first-pass decompression for distributed searches.
> 
> If that hypothesis has any validity, I'd expect you're running on
> spinning-disks rather than SSDs, is that so?
> 
> Best,
> Erick
> 
> On Tue, Jun 30, 2015 at 2:07 AM, Upayavira  wrote:
>> We need to work out why your performance is bad without optimise. What
>> version of Solr are you using? Can you confirm that your config is using
>> the TieredMergePolicy?
>> 
>> Upayavira
>> 
>> Oe, Jun 30, 2015, at 04:48 AM, Summer Shire wrote:
>>> Hi Upayavira and Erick,
>>> 
>>> There are two things we are talking about here.
>>> 
>>> First: Why am I optimizing? If I don’t our SEARCH (NOT INDEXING)
>>> performance is 100% worst.
>>> The problem lies in the number of total segments. We have to have max
>>> segments 1 or 2.
>>> I have done intensive performance related tests around number of
>>> segments, merge factor or changing the Merge policy.
>>> 
>>> Second: Solr does not perform better for me without an optimize. So now
>>> that I have to optimize the second issue
>>> is updating concurrently during an optimize. If I update when an optimize
>>> is happening the optimize takes 5 times as long as
>>> the normal optimize.
>>> 
>>> So is there any way other than creating a postOptimize hook and writing
>>> the status in a file and somehow making it available to the indexer.
>>> All of this just sounds traumatic :)
>>> 
>>> Thanks
>>> Summer
>>> 
>>> 
>>>> On Jun 29, 2015, at 5:40 AM, Erick Erickson  
>>>> wrote:
>>>> 
>>>> Steven:
>>>> 
>>>> Yes, but
>>>> 
>>>> First, here's Mike McCandles' excellent blog on segment merging:
>>>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>>>> 
>>>> I think the third animation is the TieredMergePolicy. In short, yes an
>>>> optimize will reclaim disk space. But as you update, this is done for
>>>> you anyway. About the only time optimizing is at all beneficial is
>>>> when you have a relatively static index. If you're continually
>>>> updating documents, and by that I mean replacing some existing
>>>> documents, then you'll immediately start generating "holes" in your
>>>> index.
>>>> 
>>>> And if you _do_ optimize, you wind up with a huge segment. And since
>>>> the default policy tries to merge segments of roughly the same size,
>>>> it accumulates deletes for quite a while before they merged away.
>>>> 
>>>> And if you don't update existing docs or delete docs, then there's no
>>>> wasted space anyway.
>>>> 
>>>> Summer:
>>>> 
>>>> First off, why do you care about not updating during optimizing?
>>>> There's no good reason you have to worry about that, you can freely
>>>> update while optimizing.
>>>> 
>>>> But frankly I have to agree with Upayavira that on the face of it
>>>> you're doing a lot of extra work. See above, but you optimize while
>>>> indexing, so immediately you're rather defeating the purpose.
>>>> Personally I'd only optimize relatively static indexes and, by
>>>> defi

Re: optimize status

[Summer Shire]

Hi Upayavira and Erick,

There are two things we are talking about here.

First: Why am I optimizing? If I don’t our SEARCH (NOT INDEXING) performance is 
100% worst. 
The problem lies in the number of total segments. We have to have max segments 
1 or 2. 
I have done intensive performance related tests around number of segments, 
merge factor or changing the Merge policy.

Second: Solr does not perform better for me without an optimize. So now that I 
have to optimize the second issue
is updating concurrently during an optimize. If I update when an optimize is 
happening the optimize takes 5 times as long as
the normal optimize.

So is there any way other than creating a postOptimize hook and writing the 
status in a file and somehow making it available to the indexer. 
All of this just sounds traumatic :) 

Thanks
Summer


> On Jun 29, 2015, at 5:40 AM, Erick Erickson  wrote:
> 
> Steven:
> 
> Yes, but
> 
> First, here's Mike McCandles' excellent blog on segment merging:
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
> 
> I think the third animation is the TieredMergePolicy. In short, yes an
> optimize will reclaim disk space. But as you update, this is done for
> you anyway. About the only time optimizing is at all beneficial is
> when you have a relatively static index. If you're continually
> updating documents, and by that I mean replacing some existing
> documents, then you'll immediately start generating "holes" in your
> index.
> 
> And if you _do_ optimize, you wind up with a huge segment. And since
> the default policy tries to merge segments of roughly the same size,
> it accumulates deletes for quite a while before they merged away.
> 
> And if you don't update existing docs or delete docs, then there's no
> wasted space anyway.
> 
> Summer:
> 
> First off, why do you care about not updating during optimizing?
> There's no good reason you have to worry about that, you can freely
> update while optimizing.
> 
> But frankly I have to agree with Upayavira that on the face of it
> you're doing a lot of extra work. See above, but you optimize while
> indexing, so immediately you're rather defeating the purpose.
> Personally I'd only optimize relatively static indexes and, by
> definition, you're index isn't static since the second process is just
> waiting to modify it.
> 
> Best,
> Erick
> 
> On Mon, Jun 29, 2015 at 8:15 AM, Steven White  wrote:
>> Hi Upayavira,
>> 
>> This is news to me that we should not optimize and index.
>> 
>> What about disk space saving, isn't optimization to reclaim disk space or
>> is Solr somehow does that?  Where can I read more about this?
>> 
>> I'm on Solr 5.1.0 (may switch to 5.2.1)
>> 
>> Thanks
>> 
>> Steve
>> 
>> On Mon, Jun 29, 2015 at 4:16 AM, Upayavira  wrote:
>> 
>>> I'm afraid I don't understand. You're saying that optimising is causing
>>> performance issues?
>>> 
>>> Simple solution: DO NOT OPTIMIZE!
>>> 
>>> Optimisation is very badly named. What it does is squashes all segments
>>> in your index into one segment, removing all deleted documents. It is
>>> good to get rid of deletes - in that sense the index is "optimized".
>>> However, future merges become very expensive. The best way to handle
>>> this topic is to leave it to Lucene/Solr to do it for you. Pretend the
>>> "optimize" option never existed.
>>> 
>>> This is, of course, assuming you are using something like Solr 3.5+.
>>> 
>>> Upayavira
>>> 
>>> On Mon, Jun 29, 2015, at 08:08 AM, Summer Shire wrote:
>>>> 
>>>> Have to cause of performance issues.
>>>> Just want to know if there is a way to tap into the status.
>>>> 
>>>>> On Jun 28, 2015, at 11:37 PM, Upayavira  wrote:
>>>>> 
>>>>> Bigger question, why are you optimizing? Since 3.6 or so, it generally
>>>>> hasn't been requires, even, is a bad thing.
>>>>> 
>>>>> Upayavira
>>>>> 
>>>>>> On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote:
>>>>>> Hi All,
>>>>>> 
>>>>>> I have two indexers (Independent processes ) writing to a common solr
>>>>>> core.
>>>>>> If One indexer process issued an optimize on the core
>>>>>> I want the second indexer to wait adding docs until the optimize has
>>>>>> finished.
>>>>>> 
>>>>>> Are there ways I can do this programmatically?
>>>>>> pinging the core when the optimize is happening is returning OK
>>> because
>>>>>> technically
>>>>>> solr allows you to update when an optimize is happening.
>>>>>> 
>>>>>> any suggestions ?
>>>>>> 
>>>>>> thanks,
>>>>>> Summer
>>> 

Re: optimize status

[Summer Shire]

Have to cause of performance issues. 
Just want to know if there is a way to tap into the status. 

> On Jun 28, 2015, at 11:37 PM, Upayavira  wrote:
> 
> Bigger question, why are you optimizing? Since 3.6 or so, it generally
> hasn't been requires, even, is a bad thing.
> 
> Upayavira
> 
>> On Sun, Jun 28, 2015, at 09:37 PM, Summer Shire wrote:
>> Hi All,
>> 
>> I have two indexers (Independent processes ) writing to a common solr
>> core.
>> If One indexer process issued an optimize on the core 
>> I want the second indexer to wait adding docs until the optimize has
>> finished.
>> 
>> Are there ways I can do this programmatically?
>> pinging the core when the optimize is happening is returning OK because
>> technically
>> solr allows you to update when an optimize is happening. 
>> 
>> any suggestions ?
>> 
>> thanks,
>> Summer

optimize status

[Summer Shire]

Hi All,

I have two indexers (Independent processes ) writing to a common solr core.
If One indexer process issued an optimize on the core 
I want the second indexer to wait adding docs until the optimize has finished.

Are there ways I can do this programmatically?
pinging the core when the optimize is happening is returning OK because 
technically
solr allows you to update when an optimize is happening. 

any suggestions ?

thanks,
Summer

optimize status

[Summer Shire]

Hi All,

I have two indexers (Independent processes ) writing to a common solr core.
If One indexer process issued an optimize on the core 
I want the second indexer to wait adding docs until the optimize has finished.

Are there ways I can do this programmatically?
pinging the core when the optimize is happening is returning OK because 
technically
solr allows you to update when an optimize is happening. 

any suggestions ?

thanks,
Summer

Re: invalid index version and generation

[Summer Shire]

Anyone have any suggestions on things I could try ? Does it seem like a solr 
bug ?

> On Jun 14, 2015, at 9:40 PM, Summer Shire  wrote:
> 
> Hi all,
> 
> Every time I optimize my index with maxSegment=2 after some time the 
> replication fails to get filelist for a given generation. Looks like the 
> index version and generation count gets messed up.
> (If the maxSegment=1 this never happens. I am able to successfully reproduce 
> this by optimizing with maxSegment=2. I am using solr 4.7.2)
> eg: SnapPuller- No files to download for index generation: 67
> 
> Here is what I see when I curl the commands on my terminal.
> I also tried to modify the snapPullers fetchFileList(long gen) method to get 
> indexVersion and pass that as a param but that did not help either.
> Also my slave gen/version is always smaller than master. 
> 
> What could be going on? any idea ?
> 
> Thanks,
> Summer
> 
> $ curl "http://localhost:6600/solr/main0/replication?command=indexversion 
> <http://localhost:6600/solr/main0/replication?command=indexversion>"
> 
> 
> 0 name="QTime">01434341174641 name="generation">67
> 
> 
> $ curl 
> "http://localhost:6600/solr/main0/replication?command=filelist&generation=67&indexversion=1434341174641
>  
> <http://localhost:6600/solr/main0/replication?command=filelist&generation=67&indexversion=1434341174641>"
> 
> 
> 0 name="QTime">0invalid index generation
> 

invalid index version and generation

[Summer Shire]

Hi all,

Every time I optimize my index with maxSegment=2 after some time the 
replication fails to get filelist for a given generation. Looks like the index 
version and generation count gets messed up.
(If the maxSegment=1 this never happens. I am able to successfully reproduce 
this by optimizing with maxSegment=2. I am using solr 4.7.2)
eg: SnapPuller- No files to download for index generation: 67

Here is what I see when I curl the commands on my terminal.
I also tried to modify the snapPullers fetchFileList(long gen) method to get 
indexVersion and pass that as a param but that did not help either.
Also my slave gen/version is always smaller than master. 

What could be going on? any idea ?

Thanks,
Summer

$ curl "http://localhost:6600/solr/main0/replication?command=indexversion";


00143434117464167


$ curl 
"http://localhost:6600/solr/main0/replication?command=filelist&generation=67&indexversion=1434341174641";


00invalid index generation

Re: Java 7/8 OOM exception when adding documents to httpSolrServer

[Summer Shire]

Hi Shawn,

Before I was running java 6 compiled code on java 8 when I was getting the 
exception. Now the code is compiled using java 8 and things are looking good.
I will monitor if this happens again I can share the code.

thanks,
Snehal

> On Apr 9, 2015, at 8:59 PM, Shawn Heisey  wrote:
> 
> On 4/9/2015 9:51 PM, Summer Shire wrote:
>> I was adding 1000 docs at a time but later I changed it to 10 and I am still 
>> getting the exception.
>> 
>> The reason I added the integer.max_value code was because I googled for what 
>> could be causing the OOM
>> and clearly the size of an array that worked with java6 doesn;t work with 
>> java 8.
> 
> Can you use a paste website to share your code, after redacting
> passwords, the first three octets of public IP addresses, etc?  What is
> the max heap given to Java when running your program?  Is it a
> standalone program dedicated to SolrJ, or part of something like a webapp?
> 
> Thanks,
> Shawn
> 

Re: solr 4.7.2 mergeFactor/ Merge policy issue

[Summer Shire]

Hi All,

Did anyone get a chance to look at my config and the InfoStream File ?

I am very curious to see what you think

thanks,
Summer

> On Mar 6, 2015, at 5:20 PM, Summer Shire  wrote:
> 
> Hi All,
> 
> Here’s more update on where I am at with this.
> I enabled infoStream logging and quickly figured that I need to get rid of 
> maxBufferedDocs. So Erick you 
> were absolutely right on that.
> I increased my ramBufferSize to 100MB
> and reduced maxMergeAtOnce to 3 and segmentsPerTier to 3 as well.
> My config looks like this 
> 
> 
>false
>100
> 
>
> 
>
>  3
>  3
>
>
>true
>  
> 
> I am attaching a sample infostream log file.
> In the infoStream logs though you an see how the segments keep on adding
> and it shows (just an example )
> allowedSegmentCount=10 vs count=9 (eligible count=9) tooBigCount=0
> 
> I looked at TieredMergePolicy.java to see how allowedSegmentCount is getting 
> calculated
> // Compute max allowed segs in the index
>long levelSize = minSegmentBytes;
>long bytesLeft = totIndexBytes;
>double allowedSegCount = 0;
>while(true) {
>  final double segCountLevel = bytesLeft / (double) levelSize;
>  if (segCountLevel < segsPerTier) {
>allowedSegCount += Math.ceil(segCountLevel);
>break;
>  }
>  allowedSegCount += segsPerTier;
>  bytesLeft -= segsPerTier * levelSize;
>  levelSize *= maxMergeAtOnce;
>}
>int allowedSegCountInt = (int) allowedSegCount;
> and the minSegmentBytes is calculated as follows
> // Compute total index bytes & print details about the index
>long totIndexBytes = 0;
>long minSegmentBytes = Long.MAX_VALUE;
>for(SegmentInfoPerCommit info : infosSorted) {
>  final long segBytes = size(info);
>  if (verbose()) {
>String extra = merging.contains(info) ? " [merging]" : "";
>if (segBytes >= maxMergedSegmentBytes/2.0) {
>  extra += " [skip: too large]";
>} else if (segBytes < floorSegmentBytes) {
>  extra += " [floored]";
>}
>message("  seg=" + writer.get().segString(info) + " size=" + 
> String.format(Locale.ROOT, "%.3f", segBytes/1024/1024.) + " MB" + extra);
>  }
> 
>  minSegmentBytes = Math.min(segBytes, minSegmentBytes);
>  // Accum total byte size
>  totIndexBytes += segBytes;
>}
> 
> 
> any input is welcome. 
> 
> 
> 
> 
> thanks,
> Summer
> 
> 
>> On Mar 5, 2015, at 8:11 AM, Erick Erickson  wrote:
>> 
>> I would, BTW, either just get rid of the  all together or
>> make it much higher, i.e. 10. I don't think this is really your
>> problem, but you're creating a lot of segments here.
>> 
>> But I'm kind of at a loss as to what would be different about your setup.
>> Is there _any_ chance that you have some secondary process looking at
>> your index that's maintaining open searchers? Any custom code that's
>> perhaps failing to close searchers? Is this a Unix or Windows system?
>> 
>> And just to be really clear, you _only_ seeing more segments being
>> added, right? If you're only counting files in the index directory, it's
>> _possible_ that merging is happening, you're just seeing new files take
>> the place of old ones.
>> 
>> Best,
>> Erick
>> 
>> On Wed, Mar 4, 2015 at 7:12 PM, Shawn Heisey  wrote:
>>> On 3/4/2015 4:12 PM, Erick Erickson wrote:
>>>> I _think_, but don't know for sure, that the merging stuff doesn't get
>>>> triggered until you commit, it doesn't "just happen".
>>>> 
>>>> Shot in the dark...
>>> 
>>> I believe that new segments are created when the indexing buffer
>>> (ramBufferSizeMB) fills up, even without commits.  I'm pretty sure that
>>> anytime a new segment is created, the merge policy is checked to see
>>> whether a merge is needed.
>>> 
>>> Thanks,
>>> Shawn
>>> 
> 

Re: solr 4.7.2 mergeFactor/ Merge policy issue

[Summer Shire]

Hi All,

Here’s more update on where I am at with this.
I enabled infoStream logging and quickly figured that I need to get rid of 
maxBufferedDocs. So Erick you 
were absolutely right on that.
I increased my ramBufferSize to 100MB
and reduced maxMergeAtOnce to 3 and segmentsPerTier to 3 as well.
My config looks like this 


false
100




  3
  3


true
  

I am attaching a sample infostream log file.
In the infoStream logs though you an see how the segments keep on adding
and it shows (just an example )
allowedSegmentCount=10 vs count=9 (eligible count=9) tooBigCount=0

I looked at TieredMergePolicy.java to see how allowedSegmentCount is getting 
calculated
// Compute max allowed segs in the index
long levelSize = minSegmentBytes;
long bytesLeft = totIndexBytes;
double allowedSegCount = 0;
while(true) {
  final double segCountLevel = bytesLeft / (double) levelSize;
  if (segCountLevel < segsPerTier) {
allowedSegCount += Math.ceil(segCountLevel);
break;
  }
  allowedSegCount += segsPerTier;
  bytesLeft -= segsPerTier * levelSize;
  levelSize *= maxMergeAtOnce;
}
int allowedSegCountInt = (int) allowedSegCount;
and the minSegmentBytes is calculated as follows
 // Compute total index bytes & print details about the index
long totIndexBytes = 0;
long minSegmentBytes = Long.MAX_VALUE;
for(SegmentInfoPerCommit info : infosSorted) {
  final long segBytes = size(info);
  if (verbose()) {
String extra = merging.contains(info) ? " [merging]" : "";
if (segBytes >= maxMergedSegmentBytes/2.0) {
  extra += " [skip: too large]";
} else if (segBytes < floorSegmentBytes) {
  extra += " [floored]";
}
message("  seg=" + writer.get().segString(info) + " size=" + 
String.format(Locale.ROOT, "%.3f", segBytes/1024/1024.) + " MB" + extra);
  }

  minSegmentBytes = Math.min(segBytes, minSegmentBytes);
  // Accum total byte size
  totIndexBytes += segBytes;
}


any input is welcome. 




thanks,
Summer


> On Mar 5, 2015, at 8:11 AM, Erick Erickson  wrote:
> 
> I would, BTW, either just get rid of the  all together or
> make it much higher, i.e. 10. I don't think this is really your
> problem, but you're creating a lot of segments here.
> 
> But I'm kind of at a loss as to what would be different about your setup.
> Is there _any_ chance that you have some secondary process looking at
> your index that's maintaining open searchers? Any custom code that's
> perhaps failing to close searchers? Is this a Unix or Windows system?
> 
> And just to be really clear, you _only_ seeing more segments being
> added, right? If you're only counting files in the index directory, it's
> _possible_ that merging is happening, you're just seeing new files take
> the place of old ones.
> 
> Best,
> Erick
> 
> On Wed, Mar 4, 2015 at 7:12 PM, Shawn Heisey  wrote:
>> On 3/4/2015 4:12 PM, Erick Erickson wrote:
>>> I _think_, but don't know for sure, that the merging stuff doesn't get
>>> triggered until you commit, it doesn't "just happen".
>>> 
>>> Shot in the dark...
>> 
>> I believe that new segments are created when the indexing buffer
>> (ramBufferSizeMB) fills up, even without commits.  I'm pretty sure that
>> anytime a new segment is created, the merge policy is checked to see
>> whether a merge is needed.
>> 
>> Thanks,
>> Shawn
>> 

Re: solr 4.7.2 mergeFactor/ Merge policy issue

[Summer Shire]

actually after every commit a new segment gets created. I don't see them
merging down.

what all could i do to debug this better. Hasn't anyone else tried to merge
their segments down to a specific range :) ?

On Wed, Mar 4, 2015 at 3:12 PM, Erick Erickson 
wrote:

> I _think_, but don't know for sure, that the merging stuff doesn't get
> triggered until you commit, it doesn't "just happen".
>
> Shot in the dark...
>
> Erick
>
> On Wed, Mar 4, 2015 at 1:15 PM, Summer Shire 
> wrote:
> > Hi All,
> >
> > I am using solr 4.7.2 is there a bug wrt merging the segments down ?
> >
> > I recently added the following to my solrConfig.xml
> >
> >   
> > false
> > 100
> > 1000
> > 5
> >   
> >
> >
> > But I do not see any merging of the segments happening. I saw some other
> > people have
> > the same issue but there wasn’t much info. except one suggesting to use
> > 
> >   5
> >   5
> > 
> >  class="org.apache.lucene.index.ConcurrentMergeScheduler”/>
> >
> > instead of mergeFactor.
> >
> > Thanks,
> > Summer
>

solr 4.7.2 mergeFactor/ Merge policy issue

[Summer Shire]

Hi All,

I am using solr 4.7.2 is there a bug wrt merging the segments down ?

I recently added the following to my solrConfig.xml

  
false
100
1000
5
  


But I do not see any merging of the segments happening. I saw some other
people have
the same issue but there wasn’t much info. except one suggesting to use

  5
  5

Solr 4.7.2 mergeFactor

[Summer Shire]

Hi All,

I am using solr 4.7.2 is there a bug wrt merging the segments down ? 

I recently added the following to my solrConfig.xml

  
false
100
1000
5
  


But I do not see any merging of the segments happening. I saw some other people 
have 
the same issue but there wasn’t much info. except one suggesting to use 

  5
  5

Weird Solr Replication Slave out of sync

[Summer Shire]

Hi All,

My master and slave index version and generation is the same
yet the index is not in sync because when I execute the same query 
on both master and slave I see old docs on slave which should not be there.

I also tried to fetch a specific indexversion on slave using 
command=fetchindex&indexversion=

This is very spooky because I do not get any errors on master or slave.
Also I see in the logs that the slave is polling the master every 15 mins 
I was able to find this issue only because I was looking at the specific old 
document.

Now I can manually delete the index folder on slave and restart my slave.
But I really want to find out what could be going on. Because these type of 
issues are going to 
be hard to find especially when there are on errors.

What could be happening. and how can I avoid this from happening ?


Thanks,
Summer

FirstSearcher and NewSearcher only on slaves

[Summer Shire]

HI Guys,

Is there way to specify the newSearchers and firstSearchers different for 
master vs slave ?

  

Re: Solr admin Url issues

[Summer Shire]

jetty is not running on port 80
it is running on ports that I defined for my instances in sequence.
and no I do not have apache2 reverse proxy in front :(



> On Jan 26, 2015, at 8:18 PM, Dan Davis  wrote:
> 
> Is Jetty actually running on port 80?Do you have Apache2 reverse proxy
> in front?
> 
> On Mon, Jan 26, 2015 at 11:02 PM, Summer Shire 
> wrote:
> 
>> Hi All,
>> 
>> Running solr (4.7.2) locally and hitting the admin page like this works
>> just fine http://localhost:8983/solr/ <http://localhost:8983/solr/#># <
>> http://localhost:8983/solr/#>
>> 
>> But on my deployment server my path is
>> http://example.org/jetty/MyApp/1/solr/# <
>> http://example.org/jetty/MyApp/1/solr/#>
>> Or http://example.org/jetty/MyApp/1/solr/admin/cores <
>> http://example.org/jetty/MyApp/1/solr/admin/cores> or
>> http://example.org/jetty/MyApp/1/solr/main/admin/ <
>> http://example.org/jetty/MyApp/1/solr/main/admin/>
>> 
>> the above request in a browser loads the admin page half way and then
>> spawns another request at
>> http://example.org/solr/admin/cores <http://example.org/solr/admin/cores
>>> ….
>> 
>> how can I maintain my other params such as jetty/MyApp/1/
>> 
>> btw http://example.org/jetty/MyApp/1/solr/main/select?q=*:* <
>> http://example.org/jetty/MyApp/1/solr/main/select?q=*:*> or any other
>> requesthandlers work just fine.
>> 
>> What is going on here ? any idea ?
>> 
>> thanks,
>> Summer

Solr admin Url issues

[Summer Shire]

Hi All,

Running solr (4.7.2) locally and hitting the admin page like this works just 
fine http://localhost:8983/solr/ # 
 

But on my deployment server my path is http://example.org/jetty/MyApp/1/solr/# 

Or http://example.org/jetty/MyApp/1/solr/admin/cores 
 or  
http://example.org/jetty/MyApp/1/solr/main/admin/ 


the above request in a browser loads the admin page half way and then spawns 
another request at
http://example.org/solr/admin/cores ….

how can I maintain my other params such as jetty/MyApp/1/

btw http://example.org/jetty/MyApp/1/solr/main/select?q=*:* 
 or any other 
requesthandlers work just fine.
 
What is going on here ? any idea ?

thanks,
Summer

Re: Any way to view lucene files

[Summer Shire]

Did u try  luke 47



> On Jun 6, 2014, at 11:59 PM, Aman Tandon  wrote:
> 
> I also tried with solr 4.2 and with luke version Luke 4.0.0-ALPHA
> 
> but got this error:
> java.lang.IllegalArgumentException: A SPI class of type
> org.apache.lucene.codecs.Codec with name 'Lucene42' does not exist. You
> need to add the corresponding JAR file supporting this SPI to your
> classpath.The current classpath supports the following names: [Lucene40,
> Lucene3x, SimpleText, Appending]
> 
> With Regards
> Aman Tandon
> 
> 
> On Sat, Jun 7, 2014 at 12:22 PM, Aman Tandon 
> wrote:
> 
>> My solr version is 4.8.1 and luke is 3.5
>> 
>> With Regards
>> Aman Tandon
>> 
>> 
>> On Sat, Jun 7, 2014 at 12:21 PM, Chris Collins 
>> wrote:
>> 
>>> What version of Solr / Lucene are you using?  You have to match the Luke
>>> version to the same version of Lucene.
>>> 
>>> C
 On Jun 6, 2014, at 11:42 PM, Aman Tandon  wrote:
 
 Yes  tried, but it not working at all every time i choose my index
 directory it shows me EOF past
 
 With Regards
 Aman Tandon
 
 
> On Sat, Jun 7, 2014 at 12:01 PM, Chris Collins 
 wrote:
 
> Have you tried:
> 
> https://code.google.com/p/luke/
> 
> Best
> 
> Chris
> On Jun 6, 2014, at 11:24 PM, Aman Tandon 
>>> wrote:
> 
>> Hi,
>> 
>> Is there any way so that i can view what information and which is
>>> there
> in
>> my _e.fnm, etc files. may be with the help of any application or any
> viewer
>> tool.
>> 
>> With Regards
>> Aman Tandon
>> 

Re: Solr 4.7.2 ValueSourceParser classCast exception

[Summer Shire]

I want to follow up on the ClassCastException issue with custom Plugins.
Turns out that the exception is not happening because an older version of
Solr could be some how loaded in the classpath.
I had to dig down the rabbit hole which brought me to SolrResourceLoader.

turns out that the new classLoader is created which loads all the jars from
solr.war
then it goes to look for custom plugins in the lib folder.
addToClassLoader("./lib/", null, true);
Here replaceClassLoader method is supposed to create a new classLoader that
has all the packages from solr.war and the custom jars into one but that is
not happening.
Hence if you have a class in lib folder but are extending a class  packaged
into solr.war
you will get a classCastException because of
return clazz = Class.forName(cname, true,
classLoader).asSubclass(expectedType);
line 454 in findClass method

My solution to this above problem was the add my custom jar in solr.war
this makes sure that the same classLoader loads all the jars.

That said a last year there was a patch created for 4.3 to fix the
SharedLib issue.
https://issues.apache.org/jira/browse/SOLR-4791

could this have broken the above scenarios of loading libraries from ./lib

In any case if others are facing ClassCastExceptions try adding your
library to solr.war
This was the original way of combining custom libs.


On Fri, May 23, 2014 at 1:27 PM, Jack Krupansky 
wrote:

> Are you sure that you compiled your code with the proper Solr jars so that
> the class signature (extends, implements, and constructors) matches the
> Solr 4.7.2 jars? I mean, Java is simply complaining that your class is not
> a valid value source class of the specified type.
>
> -- Jack Krupansky
>
> -Original Message- From: Summer Shire
> Sent: Friday, May 23, 2014 12:40 PM
> To: solr-user@lucene.apache.org
> Subject: Solr 4.7.2 ValueSourceParser classCast exception
>
>
> Hi All,
>
> I have my own popularity value source class
> and I let solr know about it via solrconfig.xml
>
>
>  class="mysolr.sources.PopValueSourceParser"
> />
>
> But then I get the following class cast exception
>
> I have tried to make sure there are no old Solr jar files in the classpath.
>
> Why would this be happening ?
>
> I even tried to use the lib tag to hard code the solr and solrj jars for
> 4.7.2
>
> org.apache.solr.common.SolrException: Error Instantiating
> ValueSourceParser, mysolr.sources.PopValueSourceParser failed to
> instantiate org.apache.solr.search.ValueSourceParser
> at org.apache.solr.core.SolrCore.(SolrCore.java:844)
> at org.apache.solr.core.SolrCore.(SolrCore.java:630)
> at org.apache.solr.core.CoreContainer.createFromLocal(
> CoreContainer.java:562)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:597)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.
> runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: org.apache.solr.common.SolrException: Error Instantiating
> ValueSourceParser, mysolr.sources.PopValueSourceParser failed to
> instantiate org.apache.solr.search.ValueSourceParser
> at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:552)
> at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:587)
> at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2191)
> at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2185)
> at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2218)
> at org.apache.solr.core.SolrCore.initValueSourceParsers(
> SolrCore.java:2130)
> at org.apache.solr.core.SolrCore.(SolrCore.java:765)
> ... 13 more
> Caused by: java.lang.ClassCastException: class mysolr.sources.
> PopValueSourceParser
> at java.lang.Class.asSubclass(Class.java:3018)
> at org.apache.solr.core.SolrResourceLoader.findClass(
> SolrResourceLoader.java:454)
> at org.apache.solr.core.SolrResourceLoader.findClass(
> SolrResourceLoader.java:401)
> at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:531)
> ... 19 more
> MySolr[46778:5844 0] 2014/05/22 15:47:28 717.16 MB/4.09 GB ERROR
> org.apache.solr.core.CoreContainer- 
> null:org.apache.solr.common.SolrException:
> Unable

Solr 4.7.2 ValueSourceParser classCast exception

[Summer Shire]

Hi All,

I have my own popularity value source class 
and I let solr know about it via solrconfig.xml




But then I get the following class cast exception

I have tried to make sure there are no old Solr jar files in the classpath.

Why would this be happening ? 
 
I even tried to use the lib tag to hard code the solr and solrj jars for 4.7.2

org.apache.solr.common.SolrException: Error Instantiating ValueSourceParser, 
mysolr.sources.PopValueSourceParser failed to instantiate 
org.apache.solr.search.ValueSourceParser
at org.apache.solr.core.SolrCore.(SolrCore.java:844)
at org.apache.solr.core.SolrCore.(SolrCore.java:630)
at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:562)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.solr.common.SolrException: Error Instantiating 
ValueSourceParser, mysolr.sources.PopValueSourceParser failed to instantiate 
org.apache.solr.search.ValueSourceParser
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:552)
at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:587)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2191)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2185)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2218)
at org.apache.solr.core.SolrCore.initValueSourceParsers(SolrCore.java:2130)
at org.apache.solr.core.SolrCore.(SolrCore.java:765)
... 13 more
Caused by: java.lang.ClassCastException: class 
mysolr.sources.PopValueSourceParser
at java.lang.Class.asSubclass(Class.java:3018)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:454)
at 
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:401)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:531)
... 19 more
MySolr[46778:5844 0] 2014/05/22 15:47:28 717.16 MB/4.09 GB ERROR 
org.apache.solr.core.CoreContainer- 
null:org.apache.solr.common.SolrException: Unable to create core: core1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:989)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:606)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)


Thanks,
Summer

ValueSourceParser classCast exception solr 4.7.2

[Summer Shire]

Hi All,

I have my own popularity value source class
and I let solr know about it via solrconfig.xml




But then I get the following class cast exception

I have tried to make sure there are no old Solr jar files in the classpath.

Why would this be happening ?



org.apache.solr.common.SolrException: Error Instantiating
ValueSourceParser, mysolr.sources.PopValueSourceParser failed to
instantiate org.apache.solr.search.ValueSourceParser
at org.apache.solr.core.SolrCore.(SolrCore.java:844)
at org.apache.solr.core.SolrCore.(SolrCore.java:630)
at
org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:562)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:597)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.solr.common.SolrException: Error Instantiating
ValueSourceParser, mysolr.sources.PopValueSourceParser failed to
instantiate org.apache.solr.search.ValueSourceParser
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:552)
at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:587)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2191)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2185)
at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2218)
at org.apache.solr.core.SolrCore.initValueSourceParsers(SolrCore.java:2130)
at org.apache.solr.core.SolrCore.(SolrCore.java:765)
... 13 more
Caused by: java.lang.ClassCastException: class
mysolr.sources.PopValueSourceParser
at java.lang.Class.asSubclass(Class.java:3018)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:454)
at
org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:401)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:531)
... 19 more
MySolr[46778:5844 0] 2014/05/22 15:47:28 717.16 MB/4.09 GB ERROR
org.apache.solr.core.CoreContainer-
null:org.apache.solr.common.SolrException: Unable to create core: core1
at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:989)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:606)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

solr scoring

[Summer Shire]

Hi All,

I have a scenario where for example my document titles are 
1. battle of the zords
2. joes battle
3. who was in that crazy battle.

and if the search term is [battle]
I want to penalize the second and third document score because battle was 
matched further down than in the first documents case.

One way I can do this is store the title field and then retrieve the data at 
query time -  compute the relative position of the token 
 - and use it in my custom scorer.
But this is not just extremely slow but also my index is huge.

So I am planning to index payloads along with each token which will be some 
value relative to the term position in the original string.
and then use the payload information along with any other scoring i have.

Am I thinking on the right lines? is there anything better that i could do ?

Thanks
Summer

Re: Overriding DefaultSimilarity Cast Exception 4.5.1

[Summer Shire]

Hi Shawn and Chris,
your inputs Helped I figured it out from all these links and also what Chris 
mentioned.

Thanks,
Summer

On Jan 27, 2014, at 5:54 PM, Shawn Heisey  wrote:

> On 1/27/2014 5:41 PM, Summer Shire wrote:
>> Caused by: java.lang.ClassCastException: .PayloadSimilarity 
>> cannot be cast to org.apache.lucene.search.similarities.Similarity
>>  at 
>> org.apache.solr.schema.IndexSchema$1.getSimilarity(IndexSchema.java:929)
>>  at 
>> org.apache.solr.schema.IndexSchema.getSimilarity(IndexSchema.java:257)
>>  at 
>> org.apache.solr.update.SolrIndexConfig.toIndexWriterConfig(SolrIndexConfig.java:219)
>>  at 
>> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:77)
>>  at 
>> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
>>  at 
>> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
>>  at 
>> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
>>  at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
> 
> I think there must be incorrect versions of jars in unexpected places, or 
> you're not compiling against the same version of Solr jars as the version 
> you're running, or perhaps theSolr install you've got has old versions of 
> jars hanging around, either from an old extracted .war file or elsewhere on 
> the classpath.
> 
> Here's an old thread:
> 
> http://lucene.472066.n3.nabble.com/Casting-Exception-with-Similarity-td475849.html
> 
> Thanks,
> Shawn
> 

Re: Overriding DefaultSimilarity Cast Exception 4.5.1

[Summer Shire]

Thanks Chris for a quick response.

But I verified that I only have the following jars
lucene-analyzers-common-4.5.1
lucene-core-4.5.1
lucene-queries-4.5.1
lucene-spatial-4.5.1 
solr-solrj-4.5.1

they are all the same version.

Also how can I verify the classpath ? 

Since I do not have other lucene/solr versions I assume it might just be fine.

--additional information 
also I have added a lib tag in my solrconfig.xml
  this points to the path where i 
have my jar that contains my PayloadSimilarity class
also I see in the logs that 
the file gets added to  org.apache.solr.core.SolrResourceLoader 


On Jan 27, 2014, at 5:32 PM, Chris Hostetter  wrote:

> 
> : Caused by: java.lang.ClassCastException: .PayloadSimilarity 
> : cannot be cast to org.apache.lucene.search.similarities.Similarity
> 
> If you are getting this error even though you subclass DefaultSimilarity, 
> i suspect that one or both of the following has happened:
> 
> 1) you've compiled your PayloadSimilarity against a different version of 
> solr the lucene/solr jars then what you are running with
> 2) you've got multiple versions of the lucene similarity jars in your 
> classpath at runtime.
> 
> 
> 
> -Hoss
> http://www.lucidworks.com/

Overriding DefaultSimilarity Cast Exception 4.5.1

[Summer Shire]

Hi all,

I am using Solr and Lucene 4.5.1
and I have a Payload Similarity that extends DefaultSimilarity

my schema.xml specifies the default Similarity tag


/** my similarity class**/
public class PayloadSimilarity extends DefaultSimilarity {

public PayloadSimilarity(){

}

@Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
if (payload != null) {
  return PayloadHelper.decodeFloat(payload.bytes, payload.offset);
} else {
  return 1.0f;
}
  }

}

there are no compile errors but I get runtime exception

Caused by: java.lang.ClassCastException: .PayloadSimilarity cannot 
be cast to org.apache.lucene.search.similarities.Similarity
at 
org.apache.solr.schema.IndexSchema$1.getSimilarity(IndexSchema.java:929)
at 
org.apache.solr.schema.IndexSchema.getSimilarity(IndexSchema.java:257)
at 
org.apache.solr.update.SolrIndexConfig.toIndexWriterConfig(SolrIndexConfig.java:219)
at 
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:77)
at 
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at 
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at 
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)



Why would this be happening. if Solr was not able to find my PayloadSimilarity 
class then the error message would have been different.

any inputs?

thanks,

Summer