Re: Integrate highlighting data within main search results

[Kamal Kishore Aggarwal]

any update on this guys

On Wed, May 6, 2020 at 3:39 PM Kamal Kishore Aggarwal 
wrote:

> Hi,
>
> I am using highlighting feature in solr 8.3 with default method. With
> current behaviour, main search results and highlighted results are shown in
> different blocks. Is there a way we can implemented highlighting within the
> search main results, without having to return extra block for highlighting?
>
> I believe that due to performance factor(like default limit values for
> hl.maxAnalyzedChars, hl.snippets, hl.fragsize) that highlight is returned
> as separate component. But, if someone has written custom component to
> integrate both, please share the steps. Also, please share the performance
> of it.
>
> Regards
>
> Kamal Kishore
>

Integrate highlighting data within main search results

[Kamal Kishore Aggarwal]

Hi,

I am using highlighting feature in solr 8.3 with default method. With
current behaviour, main search results and highlighted results are shown in
different blocks. Is there a way we can implemented highlighting within the
search main results, without having to return extra block for highlighting?

I believe that due to performance factor(like default limit values for
hl.maxAnalyzedChars, hl.snippets, hl.fragsize) that highlight is returned
as separate component. But, if someone has written custom component to
integrate both, please share the steps. Also, please share the performance
of it.

Regards

Kamal Kishore

Re: Recipe for moving to solr cloud without reindexing

[Kamal Kishore Aggarwal]

Hi Erick,

Is there any workaround for this?
My requirement is to move solr data from solr 6 to solr 7.

I have thought to use /replication handler to do so. But, as per the case,
the solr 6 collection has 1 shard & 1 replica and we wish to have 2 shards
and 1 replica on solr 7.
If I enable replication on collection having 2 shards, the same data from
solr 6 is copied to both the shards. i.e data becomes double.
However, if I enable replication on collection with 1 shard and later try
to SPLITSHARD, it failed with below error even after using Index Upgrade
Tool/Optimization:

*Caused by: java.lang.IllegalArgumentException: Cannot merge a segment that
has been created with major version 6 into this index which has been
created by major version 7*

Regards
Kamal

On Wed, Aug 8, 2018 at 11:58 PM Erick Erickson 
wrote:

> See: https://issues.apache.org/jira/browse/SOLR-12646
>
> On Wed, Aug 8, 2018 at 11:24 AM, Bjarke Buur Mortensen
>  wrote:
> > OK, thanks.
> >
> > As long as it's my dev box, reindexing is fine.
> > I just hope that my assumption holds, that our prod solr is 7x segments
> > only.
> >
> > Thanks again,
> > Bjarke
> >
> > 2018-08-08 20:03 GMT+02:00 Erick Erickson :
> >
> >> Bjarke:
> >>
> >> Using SPLITSHARD on an index with 6x segments just seems to not work,
> >> even outside the standalone-> cloud issue. I'll raise a JIRA.
> >> Meanwhile I think you'll have to re-index I'm afraid.
> >>
> >> Thanks for raising the issue.
> >>
> >> Erick
> >>
> >> On Wed, Aug 8, 2018 at 6:34 AM, Bjarke Buur Mortensen
> >>  wrote:
> >> > Erick,
> >> >
> >> > thanks, that is of course something I left out of the original
> question.
> >> > Our Solr is 7.1, so that should not present a problem (crossing
> fingers).
> >> >
> >> > However, on my dev box I'm trying out the steps, and here I have some
> >> > segments created with version 6 of Solr.
> >> >
> >> > After having copied data from my non-cloud solr into my
> >> > single-shard-single-replica collection and verified that Solr Cloud
> works
> >> > with this collection, I then submit the splitshard command
> >> >
> >> > http://172.17.0.4:8984
> >> > /solr/admin/collections?action=SPLITSHARD=
> >> procurement=shard1
> >> >
> >> > However, this gives me the error:
> >> > org.apache.solr.client.solrj.impl.HttpSolrClient$
> >> RemoteSolrException:Error
> >> > from server at http://172.17.0.4:8984/solr:
> >> > java.lang.IllegalArgumentException: Cannot merge a segment that has
> been
> >> > created with major version 6 into this index which has been created by
> >> > major version 7"}
> >> >
> >> > I have tried running both optimize and IndexUpgrader on the index
> before
> >> > shard splitting, but the same error still occurs.
> >> >
> >> > Any ideas as to why this happens?
> >> >
> >> > Below is an output from running IndexUpgrader, which I cannot
> decipher.
> >> > It both states that "All segments upgraded to version 7.1.0" and ''all
> >> > running merges have aborted" ¯\_(ツ)_/¯
> >> >
> >> > Thanks a lot,
> >> > Bjarke
> >> >
> >> >
> >> > ==
> >> > java -cp
> >> > /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene-
> >> backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/
> >> webapp/WEB-INF/lib/lucene-core-7.1.0.jar
> >> > org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose
> >> > /var/solr/cloud/procurement_shard1_replica_n1/data/index
> >> > IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is
> >> > "segments_4vs";
> >> > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPoli
> >> cy@721e0f4f
> >> > IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit
> "segments_4vs"
> >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint
> >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false]
> >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint
> >> > IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false
> >> > IW 0 [2018-08-08T13:00:18.273Z; main]:
> >> > dir=MMapDirectory@/var/solr/cloud/procurement_shard1_
> >> replica_n1/data/index
> >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2
> >> > index=_bhg(7.1.0):C108396
> >> > version=7.1.0
> >> > analyzer=null
> >> > ramBufferSizeMB=16.0
> >> > maxBufferedDocs=-1
> >> > mergedSegmentWarmer=null
> >> > delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
> >> > commit=null
> >> > openMode=CREATE_OR_APPEND
> >> > similarity=org.apache.lucene.search.similarities.BM25Similarity
> >> > mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1,
> >> > maxMergeCount=-1, ioThrottle=true
> >> > codec=Lucene70
> >> > infoStream=org.apache.lucene.util.PrintStreamInfoStream
> >> > mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy:
> >> maxMergeAtOnce=10,
> >> > maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0,
> floorSegmentMB=2.0,
> >> > forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0,
> >> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1)
> >> >
> 

Re: Facing issue ith MinMaxNormalizer

[Kamal Kishore Aggarwal]

Hi Christine,

Thanks for the reply.

I had identified the same issue. So, applied the below fix in the custom
jar.
 this.max = Float.parseFloat(max.replace("\"", ""));
this.min = Float.parseFloat(min.replace("\"", ""));

And, it worked. Thanks for confirming that this is an issue in solr 6.6.2

Regards
Kamal Kishore.

On Fri, Jun 14, 2019 at 2:17 PM Christine Poerschke (BLOOMBERG/ LONDON) <
cpoersc...@bloomberg.net> wrote:

> Hello Kamal Kishore,
>
> Thanks for including the Solr version alongside your question! What you
> describe sounds like the https://issues.apache.org/jira/browse/SOLR-11163
> issue which is fixed in 7.0.0 but not 6.6.2 release. The fix is a simple
> two line change to MinMaxNormalizer and perhaps one workaround could be for
> you to build a custom MinMaxNormalizer locally for use with your 6.6.2
> setup.
>
> Hope that helps.
>
> Regards,
>
> Christine
>
> From: solr-user@lucene.apache.org At: 06/12/19 12:41:51To:
> solr-user@lucene.apache.org
> Subject: Facing issue ith MinMaxNormalizer
>
> Hi All,
>
> Appreciate if someone can help.
> I am using LTR with MinMaxNormalizer in solr 6.6.2.
>
> Model.json
>
>  "class": "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
> "name": "XGBOOST-BBB-LTR-Model",
> "store":"BBB-Feature-Model",
> "features": [
> {
> "name": "TFIDF",
>  "norm":{
>  "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"0.0", "max":"1.0"}
>  }
> },
> {
> "name": "p_ratings_f",
>  "norm":{
>  "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"1.0", "max":"5.0"}
>  }
> },
> {
> "name": "p_instore_trans_cnt_f",
>  "norm":{
>  "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"1.0", "max":"209561.0" }
>  }
> },
> {
> "name": "p_reviews_f",
>  "norm":{
>  "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"0.0", "max":"58375.0"}
>  }
> }
>
>  The model got uploaded successfully, but when I reloaded the collection,
> it failed and below error is observed:
>
> Caused by: java.lang.RuntimeException: Error invoking setter setMin on
> class : org.apache.solr.ltr.norm.MinMaxNormalizer
> at
> org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1084)
> at
> org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.addModelFromMap(ManagedModelStore.java:133)
> at
> org.apache.solr.ltr.store.rest.ManagedModelStore.loadStoredModels(ManagedModelStore.java:126)
> at
> org.apache.solr.ltr.search.LTRQParserPlugin.onManagedResourceInitialized(LTRQParserPlugin.java:133)
> at
> org.apache.solr.rest.ManagedResource.notifyObserversDuringInit(ManagedResource.java:115)
> at
> org.apache.solr.rest.ManagedResource.loadManagedDataAndNotify(ManagedResource.java:91)
> at
> org.apache.solr.rest.RestManager.createManagedResource(RestManager.java:694)
> ... 41 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)
> ... 51 more
> Caused by: java.lang.NumberFormatException: For input string: ""0.0""
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
> at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
> at java.lang.Float.parseFloat(Float.java:451)
> at
> org.apache.solr.ltr.norm.MinMaxNormalizer.setMin(MinMaxNormalizer.java:58)
>
> I tried uploading the model without double quotes in param value for min
> and max, it also got failed with below error.
>
> "java.lang.IllegalArgumentException: argument type mismatch\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> 

Facing issue ith MinMaxNormalizer

[Kamal Kishore Aggarwal]

Hi All,

Appreciate if someone can help.
I am using LTR with MinMaxNormalizer in solr 6.6.2.

Model.json

 "class": "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
"name": "XGBOOST-BBB-LTR-Model",
"store":"BBB-Feature-Model",
"features": [
{
"name": "TFIDF",
 "norm":{
 "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"0.0", "max":"1.0"}
 }
},
{
"name": "p_ratings_f",
 "norm":{
 "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"1.0", "max":"5.0"}
 }
},
{
"name": "p_instore_trans_cnt_f",
 "norm":{
 "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"1.0", "max":"209561.0" }
 }
},
{
"name": "p_reviews_f",
 "norm":{
 "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
"params" : { "min":"0.0", "max":"58375.0"}
 }
}

 The model got uploaded successfully, but when I reloaded the collection,
it failed and below error is observed:

Caused by: java.lang.RuntimeException: Error invoking setter setMin on
class : org.apache.solr.ltr.norm.MinMaxNormalizer
at
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1084)
at
org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.addModelFromMap(ManagedModelStore.java:133)
at
org.apache.solr.ltr.store.rest.ManagedModelStore.loadStoredModels(ManagedModelStore.java:126)
at
org.apache.solr.ltr.search.LTRQParserPlugin.onManagedResourceInitialized(LTRQParserPlugin.java:133)
at
org.apache.solr.rest.ManagedResource.notifyObserversDuringInit(ManagedResource.java:115)
at
org.apache.solr.rest.ManagedResource.loadManagedDataAndNotify(ManagedResource.java:91)
at
org.apache.solr.rest.RestManager.createManagedResource(RestManager.java:694)
... 41 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)
... 51 more
Caused by: java.lang.NumberFormatException: For input string: ""0.0""
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.lang.Float.parseFloat(Float.java:451)
at
org.apache.solr.ltr.norm.MinMaxNormalizer.setMin(MinMaxNormalizer.java:58)

I tried uploading the model without double quotes in param value for min
and max, it also got failed with below error.

"java.lang.IllegalArgumentException: argument type mismatch\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:498)\n\tat
org.apache.solr.util.SolrPluginUtils.invokeSetters(SolrPluginUtils.java:1082)\n\tat
org.apache.solr.ltr.norm.Normalizer.getInstance(Normalizer.java:49)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.fromNormalizerMap(ManagedModelStore.java:293)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.createNormalizerFromFeatureMap(ManagedModelStore.java:276)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.fromLTRScoringModelMap(ManagedModelStore.java:230)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.addModelFromMap(ManagedModelStore.java:133)\n\tat
org.apache.solr.ltr.store.rest.ManagedModelStore.applyUpdatesToManagedData(ManagedModelStore.java:162)\n\tat

Also, I can see on the collection that there is escape in double quotes in
model on model api (s/s below).

[image: image.png]


Regards
Kamal Kishore

LTR: Normalize Feature Weights

[Kamal Kishore Aggarwal]

Hi,

Is there a way to normalize the value of fieldValueFeature
and OriginalScoreFeature features within some range i.e 0-1.

Lets suppose I have 4 products with some field values, I wish to normalize
weight within 0 and 1 using func (val-min)/(max-min).

Product FieldValue Normalized Value
P1 4 1
P2 3 0.6
P3 2 0.3
P4 1 0
P5 - -1

If the product does not contain the field value, make feature value as -1
(some static).

I tried to use the scale function, but since scale function works on the
whole index, so it will not be relevant for our case. If you multiple
function here, performance will be impacted.
I have seen solr ltr source code and there is a normalized function, but
not sure how to implement it in our case.

Regards
Kamal

Re: Solr LTR model Performance Issues

[Kamal Kishore Aggarwal]

Hi,

I made change in the model by making the LTRScoringModel as immutable and
cache hashCode calculation. The response time improved a lot after the
change.

http://lucene.472066.n3.nabble.com/jira-Updated-SOLR-12688-LTR-Multiple-performance-fixes-pure-DocValues-support-for-FieldValueFeature-td4404254.html


On Sat, Apr 6, 2019 at 12:22 PM Jörn Franke  wrote:

> It is a little bit difficult to say, because it could be also the business
> logic in the query execution. What is your performance baseline, ie if you
> just execute one query for each of the models?
> How fast should it be? Do you have really 10 or more concurrent users, or
> users that fire up queries at exactly the same time?
>
> Can you please monitor CPU and memory?
>
> > Am 05.04.2019 um 21:42 schrieb Kamal Kishore Aggarwal <
> kkroyal@gmail.com>:
> >
> > Hi,
> >
> > Any update on this?
> > Is this model running in multi threaded mode or is there is any scope to
> do
> > this. Please let me know.
> >
> > Regards
> > Kamal
> >
> > On Sat, Mar 23, 2019 at 10:35 AM Kamal Kishore Aggarwal <
> > kkroyal@gmail.com> wrote:
> >
> >> HI Jörn Franke,
> >>
> >> Thanks for the quick reply.
> >>
> >> I have performed the jmeter load testing on one of the server for Linear
> >> vs Multipleadditive tree model. We are using lucidworks fusion.
> >> There is some business logic in the query pipeline followed by main solr
> >> ltr query. This is the total time taken by query pipeline.
> >> Below are the response time:
> >>
> >> # of Threads Ramup Period Loop Count Type Total Requests Average
> Response
> >> Time (ms)
> >> Iteration 1 Iteration 2 Iteration 3
> >> 10 1 10 Linear Model  100 2038 1998 1975
> >> 25 1 10 Linear Model  250 4329 3961 3726
> >>
> >> 10 1 10 MultiAdditive Model 100 12721 12631 12567
> >> 25 1 10 MultiAdditive Model 250 27924 31420 30758
> >> # of docs: 500K and Indexing size is 10 GB.
> >>
> >> As of now, I did not checked the CPU or memory usage, but did not
> observed
> >> any errors during jmeter load test.
> >>
> >> Let me know if any other information is required.
> >>
> >> Regards
> >> Kamal
> >>
> >>
> >> <
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> I’m
> >> protected online with Avast Free Antivirus. Get it here — it’s free
> >> forever.
> >> <
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> >> <#m_-1438210790161476832_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>
> >>> On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke 
> wrote:
> >>>
> >>> Can you share the time needed of the two models? How many documents?
> What
> >>> is your loading pipeline? Have you observed cpu/memory?
> >>>
> >>>> Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
> >>> kkroyal@gmail.com>:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am trying to use LTR with solr 6.6.2.There are different types of
> >>> model
> >>>> like Linear Model, Multiple Additive Trees Model and Neural Network
> >>> Model.
> >>>>
> >>>> I have tried using Linear & Multiadditive model and compared the
> >>>> performance of results. There is a major difference in response time
> >>>> between the 2 models. I am observing that Multiadditive model is
> taking
> >>> way
> >>>> higher time than linear model.
> >>>>
> >>>> Is there a way we can improve the performance here.
> >>>>
> >>>> Note: The size of Multiadditive model is 136 MB.
> >>>>
> >>>> Regards
> >>>> Kamal Kishore
> >>>>
> >>>> <
> >>>
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >>>>
> >>>> I’m
> >>>> protected online with Avast Free Antivirus. Get it here — it’s free
> >>> forever.
> >>>> <
> >>>
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >>>>
> >>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>>
> >>
>

Re: Solr LTR model Performance Issues

[Kamal Kishore Aggarwal]

Hi,

Any update on this?
Is this model running in multi threaded mode or is there is any scope to do
this. Please let me know.

Regards
Kamal

On Sat, Mar 23, 2019 at 10:35 AM Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> HI Jörn Franke,
>
> Thanks for the quick reply.
>
> I have performed the jmeter load testing on one of the server for Linear
> vs Multipleadditive tree model. We are using lucidworks fusion.
> There is some business logic in the query pipeline followed by main solr
> ltr query. This is the total time taken by query pipeline.
> Below are the response time:
>
> # of Threads Ramup Period Loop Count Type Total Requests Average Response
> Time (ms)
> Iteration 1 Iteration 2 Iteration 3
> 10 1 10 Linear Model  100 2038 1998 1975
> 25 1 10 Linear Model  250 4329 3961 3726
>
> 10 1 10 MultiAdditive Model 100 12721 12631 12567
> 25 1 10 MultiAdditive Model 250 27924 31420 30758
> # of docs: 500K and Indexing size is 10 GB.
>
> As of now, I did not checked the CPU or memory usage, but did not observed
> any errors during jmeter load test.
>
> Let me know if any other information is required.
>
> Regards
> Kamal
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
>  I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
> <#m_-1438210790161476832_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke  wrote:
>
>> Can you share the time needed of the two models? How many documents? What
>> is your loading pipeline? Have you observed cpu/memory?
>>
>> > Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
>> kkroyal@gmail.com>:
>> >
>> > Hi,
>> >
>> > I am trying to use LTR with solr 6.6.2.There are different types of
>> model
>> > like Linear Model, Multiple Additive Trees Model and Neural Network
>> Model.
>> >
>> > I have tried using Linear & Multiadditive model and compared the
>> > performance of results. There is a major difference in response time
>> > between the 2 models. I am observing that Multiadditive model is taking
>> way
>> > higher time than linear model.
>> >
>> > Is there a way we can improve the performance here.
>> >
>> > Note: The size of Multiadditive model is 136 MB.
>> >
>> > Regards
>> > Kamal Kishore
>> >
>> > <
>> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>> >
>> > I’m
>> > protected online with Avast Free Antivirus. Get it here — it’s free
>> forever.
>> > <
>> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
>> >
>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>

Re: Model type does not exist MultipleAdditiveTreesModel

[Kamal Kishore Aggarwal]

Hi Roee,

It looks the error is due to blank feature param value in the json.

"name" : "my",
   "features":[],
   "params" : {

I have observed that many a times solr ltr returns generic error that 'Model
type does not
exist', but later actually found to be an issue with json. Just wanted to
share my experience.

Regards
Kamal

On Thu, May 31, 2018 at 4:07 PM Roee T  wrote:

> Hi all,
> I'm trying to upload the most simple model to solr 7.3.1 and i get an
> error:
>
> the model:
>
> {
>"class" : "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
>"name" : "my",
>"features":[],
>"params" : {
>"trees" : [
>{
>"weight" : 1,
>"root" : {
>"value" : -10
>}} ]}}
>
> The error:
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","java.lang.IllegalArgumentException"],
> "msg":"org.apache.solr.ltr.model.ModelException: Model type does not
> exist org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
> "code":400}}
>
>
> I inserted the configurations to solrconfig.xml like
>regex=".*\.jar" />
> and started solr using   -Dsolr.ltr.enabled=true
>
> please help me
> Thanks you all ;)
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>

Re: Solr LTR model Performance Issues

[Kamal Kishore Aggarwal]

HI Jörn Franke,

Thanks for the quick reply.

I have performed the jmeter load testing on one of the server for Linear vs
Multipleadditive tree model. We are using lucidworks fusion.
There is some business logic in the query pipeline followed by main solr
ltr query. This is the total time taken by query pipeline.
Below are the response time:

# of Threads Ramup Period Loop Count Type Total Requests Average Response
Time (ms)
Iteration 1 Iteration 2 Iteration 3
10 1 10 Linear Model  100 2038 1998 1975
25 1 10 Linear Model  250 4329 3961 3726

10 1 10 MultiAdditive Model 100 12721 12631 12567
25 1 10 MultiAdditive Model 250 27924 31420 30758
# of docs: 500K and Indexing size is 10 GB.

As of now, I did not checked the CPU or memory usage, but did not observed
any errors during jmeter load test.

Let me know if any other information is required.

Regards
Kamal

<https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
I’m
protected online with Avast Free Antivirus. Get it here — it’s free forever.
<https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Fri, Mar 22, 2019 at 5:13 PM Jörn Franke  wrote:

> Can you share the time needed of the two models? How many documents? What
> is your loading pipeline? Have you observed cpu/memory?
>
> > Am 22.03.2019 um 12:01 schrieb Kamal Kishore Aggarwal <
> kkroyal@gmail.com>:
> >
> > Hi,
> >
> > I am trying to use LTR with solr 6.6.2.There are different types of model
> > like Linear Model, Multiple Additive Trees Model and Neural Network
> Model.
> >
> > I have tried using Linear & Multiadditive model and compared the
> > performance of results. There is a major difference in response time
> > between the 2 models. I am observing that Multiadditive model is taking
> way
> > higher time than linear model.
> >
> > Is there a way we can improve the performance here.
> >
> > Note: The size of Multiadditive model is 136 MB.
> >
> > Regards
> > Kamal Kishore
> >
> > <
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> > I’m
> > protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> > <
> https://www.avast.com/en-in/recommend?utm_medium=email_source=link_campaign=sig-email_content=webmail_term=default3=d4ef6ef9-b8d1-40b8-96ac-2354fd69483b
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Solr LTR model Performance Issues

[Kamal Kishore Aggarwal]

Hi,

I am trying to use LTR with solr 6.6.2.There are different types of model
like Linear Model, Multiple Additive Trees Model and Neural Network Model.

I have tried using Linear & Multiadditive model and compared the
performance of results. There is a major difference in response time
between the 2 models. I am observing that Multiadditive model is taking way
higher time than linear model.

Is there a way we can improve the performance here.

Note: The size of Multiadditive model is 136 MB.

Regards
Kamal Kishore


I’m
protected online with Avast Free Antivirus. Get it here — it’s free forever.

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: LTR feature based on other collection data

[Kamal Kishore Aggarwal]

any suggestions ?

Thanks in advance.

On Tue, Feb 26, 2019 at 6:22 PM Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> I looks to me that I can modify the *SolrFeature *class, but dont know
> how to create IndexSearcher and SolrQueryRequest params as per the new
> request and second collection.
>
> @Override
>   public FeatureWeight createWeight(*IndexSearcher searcher*, boolean
> needsScores,
>   *SolrQueryRequest request*, Query originalQuery,
> Map efi)
>   throws IOException {
> return new SolrFeatureWeight(searcher, request, originalQuery, efi);
>   }
>
> Regards
> Kamal
>
>
> On Tue, Feb 26, 2019 at 12:34 PM Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
>> Hi,
>>
>> I am working on LTR using solr 6.6.2. I am working on custom feature
>> creation. I am able to create few custom features as per our requirement.
>>
>> But, there are certain features, for which the data is stored in other
>> collection. Data like count of clicks, last date when the product was
>> ordered, etc. These type of information is stored in another collection and
>> we are not planning to put this info. in first collection.
>>
>> Now, we need to use the data in other collection to generate the score of
>> the document  in LTR. We are open to develop custom components as well.
>>
>> Is there a way, we can modify our query using some join. But, we know
>> join is expensive.
>>
>> Please suggest. Thanks in advance.
>>
>> Regards
>> Kamal Kishore
>>
>

Re: LTR feature based on other collection data

[Kamal Kishore Aggarwal]

I looks to me that I can modify the *SolrFeature *class, but dont know how
to create IndexSearcher and SolrQueryRequest params as per the new request
and second collection.

@Override
  public FeatureWeight createWeight(*IndexSearcher searcher*, boolean
needsScores,
  *SolrQueryRequest request*, Query originalQuery, Map
efi)
  throws IOException {
return new SolrFeatureWeight(searcher, request, originalQuery, efi);
  }

Regards
Kamal


On Tue, Feb 26, 2019 at 12:34 PM Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi,
>
> I am working on LTR using solr 6.6.2. I am working on custom feature
> creation. I am able to create few custom features as per our requirement.
>
> But, there are certain features, for which the data is stored in other
> collection. Data like count of clicks, last date when the product was
> ordered, etc. These type of information is stored in another collection and
> we are not planning to put this info. in first collection.
>
> Now, we need to use the data in other collection to generate the score of
> the document  in LTR. We are open to develop custom components as well.
>
> Is there a way, we can modify our query using some join. But, we know join
> is expensive.
>
> Please suggest. Thanks in advance.
>
> Regards
> Kamal Kishore
>

LTR feature based on other collection data

[Kamal Kishore Aggarwal]

Hi,

I am working on LTR using solr 6.6.2. I am working on custom feature
creation. I am able to create few custom features as per our requirement.

But, there are certain features, for which the data is stored in other
collection. Data like count of clicks, last date when the product was
ordered, etc. These type of information is stored in another collection and
we are not planning to put this info. in first collection.

Now, we need to use the data in other collection to generate the score of
the document  in LTR. We are open to develop custom components as well.

Is there a way, we can modify our query using some join. But, we know join
is expensive.

Please suggest. Thanks in advance.

Regards
Kamal Kishore

Re: Solr Stream vs Export Request Handlers

[Kamal Kishore Aggarwal]

Any update on this.

Regards
Kamal

On Thu, Oct 18, 2018 at 11:50 AM Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi,
>
> Thanks again Joel for your reply. I have noted your suggestions.
>
> I observed one more thing while using solrj to fetch the data using
> /stream with export and direct /export. The solr QTime is almost same,
> however elapsed time(total time) to fetch response in streaming with export
> is better than direct /export (Streaming export taking 30% less than
> /export).
>
> Is this also expected ?
>
> Regards
> Kamal Kishore
>
>
>
> On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein  wrote:
>
>> Yes this is correct. But keep in mind Streaming Expression has a wide
>> range
>> of features that have nothing at all to do with the export handler. In
>> general with Streaming Expressions you want to find the functions that get
>> the job done using the least amount of work. The /export handler is often
>> not the best choice. You'll want to read through the various streaming
>> expressions to see if they might be more efficient for your use case.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
>> kkroyal@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > After I performed the test on my data, I found out that direct /export
>> and
>> > streaming expression with export, both are giving almost same response
>> > time. This was also pointed out by *Jan Høydahl* in his reply.
>> >
>> > Also, the documentation says export feature uses stream sorting
>> technique
>> > and streaming expression also uses steam technique. So, are they
>> internally
>> > works in same fashion. Please confirm.
>> >
>> > Regards
>> > Kamal Kishore
>> >
>> >
>> >
>> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
>> > kkroyal@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Thanks Jan & Joel.
>> > >
>> > > Though I will evaluate the performance over my data, but based on your
>> > > experience, which one of the two is better in performance ?. Please
>> > suggest
>> > >
>> > > Yeah, I know export does not get the data from all shards, but we can
>> > > write code to aggregate the data from all shards. But only if export
>> is
>> > > better over stream.
>> > >
>> > > Thanks
>> > > Kamal Kishore
>> > >
>> > >
>> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein 
>> > > wrote:
>> > >
>> > >> The export handler does not do distributed search. So if you have a
>> > >> multi-shard collection you may have to use Streaming Expressions to
>> get
>> > >> exports from all shards.
>> > >>
>> > >>
>> > >> Joel Bernstein
>> > >> http://joelsolr.blogspot.com/
>> > >>
>> > >>
>> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl 
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > Yes, you can choose which to use, it should give  you about same
>> > result.
>> > >> > If you already work with the Solr search API it would be the
>> easiest
>> > for
>> > >> > you to consume /export as you don't need to learn the new syntax
>> and
>> > >> parse
>> > >> > the Tuple response. However, if you need to do stuff with the docs
>> as
>> > >> you
>> > >> > stream them from Solr, then streaming expressions lets you enrich
>> the
>> > >> docs,
>> > >> > modify, join etc on the fly.
>> > >> >
>> > >> > PS: When the /export docs says it uses a streaming tecnique, it
>> does
>> > NOT
>> > >> > mean that it has uses the solr feature streaming expressions :)
>> > >> >
>> > >> > --
>> > >> > Jan Høydahl, search solution architect
>> > >> > Cominvent AS - www.cominvent.com
>> > >> >
>> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > >> > kkroyal@gmail.com>:
>> > >> > >
>> > >> > > Hi,
>> > 

Re: Solr Stream vs Export Request Handlers

[Kamal Kishore Aggarwal]

Hi,

Thanks again Joel for your reply. I have noted your suggestions.

I observed one more thing while using solrj to fetch the data using /stream
with export and direct /export. The solr QTime is almost same, however
elapsed time(total time) to fetch response in streaming with export is
better than direct /export (Streaming export taking 30% less than /export).

Is this also expected ?

Regards
Kamal Kishore



On Tue, Oct 16, 2018 at 3:21 AM Joel Bernstein  wrote:

> Yes this is correct. But keep in mind Streaming Expression has a wide range
> of features that have nothing at all to do with the export handler. In
> general with Streaming Expressions you want to find the functions that get
> the job done using the least amount of work. The /export handler is often
> not the best choice. You'll want to read through the various streaming
> expressions to see if they might be more efficient for your use case.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Oct 15, 2018 at 12:05 PM Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
> > Hi,
> >
> > After I performed the test on my data, I found out that direct /export
> and
> > streaming expression with export, both are giving almost same response
> > time. This was also pointed out by *Jan Høydahl* in his reply.
> >
> > Also, the documentation says export feature uses stream sorting technique
> > and streaming expression also uses steam technique. So, are they
> internally
> > works in same fashion. Please confirm.
> >
> > Regards
> > Kamal Kishore
> >
> >
> >
> > On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal <
> > kkroyal@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Thanks Jan & Joel.
> > >
> > > Though I will evaluate the performance over my data, but based on your
> > > experience, which one of the two is better in performance ?. Please
> > suggest
> > >
> > > Yeah, I know export does not get the data from all shards, but we can
> > > write code to aggregate the data from all shards. But only if export is
> > > better over stream.
> > >
> > > Thanks
> > > Kamal Kishore
> > >
> > >
> > > On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein 
> > > wrote:
> > >
> > >> The export handler does not do distributed search. So if you have a
> > >> multi-shard collection you may have to use Streaming Expressions to
> get
> > >> exports from all shards.
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >>
> > >> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl 
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Yes, you can choose which to use, it should give  you about same
> > result.
> > >> > If you already work with the Solr search API it would be the easiest
> > for
> > >> > you to consume /export as you don't need to learn the new syntax and
> > >> parse
> > >> > the Tuple response. However, if you need to do stuff with the docs
> as
> > >> you
> > >> > stream them from Solr, then streaming expressions lets you enrich
> the
> > >> docs,
> > >> > modify, join etc on the fly.
> > >> >
> > >> > PS: When the /export docs says it uses a streaming tecnique, it does
> > NOT
> > >> > mean that it has uses the solr feature streaming expressions :)
> > >> >
> > >> > --
> > >> > Jan Høydahl, search solution architect
> > >> > Cominvent AS - www.cominvent.com
> > >> >
> > >> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > >> > kkroyal@gmail.com>:
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I have a requirement to fetch all data from a collection. One way
> is
> > >> to
> > >> > use
> > >> > > streaming expression and other way is to use export.
> > >> > >
> > >> > > Streaming expression documentation says *streaming functions are
> > >> designed
> > >> > > to work with entire result sets rather then the top N results like
> > >> normal
> > >> > > search. This is supported by the /export handler.*
> > >> > >
> > >> > > Also, Export handler documentation says *this feature uses a
> stream
> > >> > sorting
> > >> > > technique that begins to send records within milliseconds and
> > >> continues
> > >> > to
> > >> > > stream results until the entire result set has been sorted and
> > >> exported.*
> > >> > >
> > >> > > These two statements concludes to me that for fetching entire
> > results
> > >> > > streaming expressions uses export handler and export handler uses
> > >> stream,
> > >> > > so, whether I use streaming expression or export handler, they are
> > >> > > internally same and would have same performance. I am correct over
> > >> here
> > >> > to
> > >> > > say so ?
> > >> > >
> > >> > >
> > >> > > Ref Links:
> > >> > >
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > >> > >
> https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >> > >
> > >> > >
> > >> > > Regards
> > >> > > Kamal Kishore
> > >> >
> > >> >
> > >>
> > >
> >
>

Re: Solr Stream vs Export Request Handlers

[Kamal Kishore Aggarwal]

Hi,

After I performed the test on my data, I found out that direct /export and
streaming expression with export, both are giving almost same response
time. This was also pointed out by *Jan Høydahl* in his reply.

Also, the documentation says export feature uses stream sorting technique
and streaming expression also uses steam technique. So, are they internally
works in same fashion. Please confirm.

Regards
Kamal Kishore



On Tue, Oct 2, 2018 at 5:51 PM Kamal Kishore Aggarwal 
wrote:

> Hi,
>
> Thanks Jan & Joel.
>
> Though I will evaluate the performance over my data, but based on your
> experience, which one of the two is better in performance ?. Please suggest
>
> Yeah, I know export does not get the data from all shards, but we can
> write code to aggregate the data from all shards. But only if export is
> better over stream.
>
> Thanks
> Kamal Kishore
>
>
> On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein 
> wrote:
>
>> The export handler does not do distributed search. So if you have a
>> multi-shard collection you may have to use Streaming Expressions to get
>> exports from all shards.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl 
>> wrote:
>>
>> > Hi,
>> >
>> > Yes, you can choose which to use, it should give  you about same result.
>> > If you already work with the Solr search API it would be the easiest for
>> > you to consume /export as you don't need to learn the new syntax and
>> parse
>> > the Tuple response. However, if you need to do stuff with the docs as
>> you
>> > stream them from Solr, then streaming expressions lets you enrich the
>> docs,
>> > modify, join etc on the fly.
>> >
>> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
>> > mean that it has uses the solr feature streaming expressions :)
>> >
>> > --
>> > Jan Høydahl, search solution architect
>> > Cominvent AS - www.cominvent.com
>> >
>> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
>> > kkroyal@gmail.com>:
>> > >
>> > > Hi,
>> > >
>> > > I have a requirement to fetch all data from a collection. One way is
>> to
>> > use
>> > > streaming expression and other way is to use export.
>> > >
>> > > Streaming expression documentation says *streaming functions are
>> designed
>> > > to work with entire result sets rather then the top N results like
>> normal
>> > > search. This is supported by the /export handler.*
>> > >
>> > > Also, Export handler documentation says *this feature uses a stream
>> > sorting
>> > > technique that begins to send records within milliseconds and
>> continues
>> > to
>> > > stream results until the entire result set has been sorted and
>> exported.*
>> > >
>> > > These two statements concludes to me that for fetching entire results
>> > > streaming expressions uses export handler and export handler uses
>> stream,
>> > > so, whether I use streaming expression or export handler, they are
>> > > internally same and would have same performance. I am correct over
>> here
>> > to
>> > > say so ?
>> > >
>> > >
>> > > Ref Links:
>> > >
>> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
>> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
>> > >
>> > >
>> > > Regards
>> > > Kamal Kishore
>> >
>> >
>>
>

Re: Solr Stream vs Export Request Handlers

[Kamal Kishore Aggarwal]

Hi,

Thanks Jan & Joel.

Though I will evaluate the performance over my data, but based on your
experience, which one of the two is better in performance ?. Please suggest

Yeah, I know export does not get the data from all shards, but we can write
code to aggregate the data from all shards. But only if export is better
over stream.

Thanks
Kamal Kishore


On Thu, Sep 27, 2018 at 11:04 PM Joel Bernstein  wrote:

> The export handler does not do distributed search. So if you have a
> multi-shard collection you may have to use Streaming Expressions to get
> exports from all shards.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Thu, Sep 27, 2018 at 4:32 AM Jan Høydahl  wrote:
>
> > Hi,
> >
> > Yes, you can choose which to use, it should give  you about same result.
> > If you already work with the Solr search API it would be the easiest for
> > you to consume /export as you don't need to learn the new syntax and
> parse
> > the Tuple response. However, if you need to do stuff with the docs as you
> > stream them from Solr, then streaming expressions lets you enrich the
> docs,
> > modify, join etc on the fly.
> >
> > PS: When the /export docs says it uses a streaming tecnique, it does NOT
> > mean that it has uses the solr feature streaming expressions :)
> >
> > --
> > Jan Høydahl, search solution architect
> > Cominvent AS - www.cominvent.com
> >
> > > 27. sep. 2018 kl. 09:07 skrev Kamal Kishore Aggarwal <
> > kkroyal@gmail.com>:
> > >
> > > Hi,
> > >
> > > I have a requirement to fetch all data from a collection. One way is to
> > use
> > > streaming expression and other way is to use export.
> > >
> > > Streaming expression documentation says *streaming functions are
> designed
> > > to work with entire result sets rather then the top N results like
> normal
> > > search. This is supported by the /export handler.*
> > >
> > > Also, Export handler documentation says *this feature uses a stream
> > sorting
> > > technique that begins to send records within milliseconds and continues
> > to
> > > stream results until the entire result set has been sorted and
> exported.*
> > >
> > > These two statements concludes to me that for fetching entire results
> > > streaming expressions uses export handler and export handler uses
> stream,
> > > so, whether I use streaming expression or export handler, they are
> > > internally same and would have same performance. I am correct over here
> > to
> > > say so ?
> > >
> > >
> > > Ref Links:
> > >
> > > https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
> > > https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html
> > >
> > >
> > > Regards
> > > Kamal Kishore
> >
> >
>

Solr Stream vs Export Request Handlers

[Kamal Kishore Aggarwal]

Hi,

I have a requirement to fetch all data from a collection. One way is to use
streaming expression and other way is to use export.

Streaming expression documentation says *streaming functions are designed
to work with entire result sets rather then the top N results like normal
search. This is supported by the /export handler.*

Also, Export handler documentation says *this feature uses a stream sorting
technique that begins to send records within milliseconds and continues to
stream results until the entire result set has been sorted and exported.*

These two statements concludes to me that for fetching entire results
streaming expressions uses export handler and export handler uses stream,
so, whether I use streaming expression or export handler, they are
internally same and would have same performance. I am correct over here to
say so ?


Ref Links:

https://lucene.apache.org/solr/guide/6_6/streaming-expressions.html
https://lucene.apache.org/solr/guide/6_6/exporting-result-sets.html


Regards
Kamal Kishore

Re: Solr Default query parser

[Kamal Kishore Aggarwal]

Thanks Jason and Shawn.

It's clear now.


Regards
Kamal


On Tue, Jun 26, 2018, 6:12 PM Jason Gerlowski  wrote:

> The "Standard Query Parser" _is_ the lucene query parser.  They're the
> same parser.  As Shawn pointed out above, they're also the default, so
> if you don't specify any defType, they will be used.  Though if you
> want to be explicit and specify it anyway, the value is defType=lucene
>
> Jason
> On Mon, Jun 25, 2018 at 1:05 PM Kamal Kishore Aggarwal
>  wrote:
> >
> > Hi Shawn,
> >
> > Thanks for the reply.
> >
> > If "lucene" is the default query parser, then how can we specify Standard
> > Query Parser(QP) in the query.
> >
> > Dismax QP can be specified by defType=dismax and Extended Dismax Qp by
> > defType=edismax, how about for declaration of Standard QP.
> >
> > Regards
> > Kamal
> >
> > On Wed, Jun 6, 2018 at 9:41 PM, Shawn Heisey 
> wrote:
> >
> > > On 6/6/2018 9:52 AM, Kamal Kishore Aggarwal wrote:
> > > >> What is the default query parser (QP) for solr.
> > > >>
> > > >> While I was reading about this, I came across two links which looks
> > > >> ambiguous to me. It's not clear to me whether Standard is the
> default
> > > QP or
> > > >> Lucene is the default QP or they are same. Below is the screenshot
> and
> > > >> links which are confusing me.
> > >
> > > The default query parser in Solr has the name "lucene".  This query
> > > parser, which is part of Solr, deals with Lucene query syntax.
> > >
> > > The most recent documentation states this clearly right after the table
> > > of contents:
> > >
> > >
> https://lucene.apache.org/solr/guide/7_3/the-standard-query-parser.html
> > >
> > > It is highly unlikely that the 6.6 documentation will receive any
> > > changes, unless serious errors are found in it.  The omission of this
> > > piece of information will not be seen as a serious error.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
>

Re: Solr Default query parser

[Kamal Kishore Aggarwal]

Hi Shawn,

Thanks for the reply.

If "lucene" is the default query parser, then how can we specify Standard
Query Parser(QP) in the query.

Dismax QP can be specified by defType=dismax and Extended Dismax Qp by
defType=edismax, how about for declaration of Standard QP.

Regards
Kamal

On Wed, Jun 6, 2018 at 9:41 PM, Shawn Heisey  wrote:

> On 6/6/2018 9:52 AM, Kamal Kishore Aggarwal wrote:
> >> What is the default query parser (QP) for solr.
> >>
> >> While I was reading about this, I came across two links which looks
> >> ambiguous to me. It's not clear to me whether Standard is the default
> QP or
> >> Lucene is the default QP or they are same. Below is the screenshot and
> >> links which are confusing me.
>
> The default query parser in Solr has the name "lucene".  This query
> parser, which is part of Solr, deals with Lucene query syntax.
>
> The most recent documentation states this clearly right after the table
> of contents:
>
> https://lucene.apache.org/solr/guide/7_3/the-standard-query-parser.html
>
> It is highly unlikely that the 6.6 documentation will receive any
> changes, unless serious errors are found in it.  The omission of this
> piece of information will not be seen as a serious error.
>
> Thanks,
> Shawn
>
>

Re: Solr Default query parser

[Kamal Kishore Aggarwal]

[Correcting the subject]

On Wed, Jun 6, 2018 at 2:37 PM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi Guys,
>
> What is the default query parser (QP) for solr.
>
> While I was reading about this, I came across two links which looks
> ambiguous to me. It's not clear to me whether Standard is the default QP or
> Lucene is the default QP or they are same. Below is the screenshot and
> links which are confusing me.
>
> https://lucene.apache.org/solr/guide/6_6/the-standard-query-parser.html
>
> https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html
>
> Please suggest. Thanks in advance.
>
>
> Regards
> Kamal Kishore
>

Sole Default query parser

[Kamal Kishore Aggarwal]

Hi Guys,

What is the default query parser (QP) for solr.

While I was reading about this, I came across two links which looks
ambiguous to me. It's not clear to me whether Standard is the default QP or
Lucene is the default QP or they are same. Below is the screenshot and
links which are confusing me.

https://lucene.apache.org/solr/guide/6_6/the-standard-query-parser.html

https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html

Please suggest. Thanks in advance.


Regards
Kamal Kishore

Re: Clob transformer not working in DIH

[Kamal Kishore Aggarwal]

Any help would be appreciated.

On 12-Dec-2016 1:20 PM, "Kamal Kishore Aggarwal" <kkroyal@gmail.com>
wrote:

> Any help guys ...
>
> On 09-Dec-2016 1:05 PM, "Kamal Kishore Aggarwal" <kkroyal@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using solr 5.4.1. Here I am using dataimport handler to index data
>> with SQL Server.
>>
>> I am using CLOB transformer to convert clob value to string. Indexing is
>> working fine but clob transformation is not working. Expected string value
>> is not coming for clob column. There is no error or exception coming in log.
>>
>> Here is the configuration:
>>
>> 
>> > url="jdbc:sqlserver://localhost;databaseName=Dictionary;"
>>  user="sa"password="" batchSize="5" />
>>
>> 
>>
>> > transformer="ClobTransformer">
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> I tried using RegexTransformer, it worked. But ClobTransformer is not
>> working. Please assist.
>>
>> Regards
>> Kamal
>>
>

Re: Clob transformer not working in DIH

[Kamal Kishore Aggarwal]

Any help guys ...

On 09-Dec-2016 1:05 PM, "Kamal Kishore Aggarwal" <kkroyal@gmail.com>
wrote:

> Hi,
>
> I am using solr 5.4.1. Here I am using dataimport handler to index data
> with SQL Server.
>
> I am using CLOB transformer to convert clob value to string. Indexing is
> working fine but clob transformation is not working. Expected string value
> is not coming for clob column. There is no error or exception coming in log.
>
> Here is the configuration:
>
> 
>  url="jdbc:sqlserver://localhost;databaseName=Dictionary;"
>  user="sa"password="" batchSize="5" />
>
> 
>
>  transformer="ClobTransformer">
>
> 
>
> 
>
> 
>
> 
>
> 
>
> I tried using RegexTransformer, it worked. But ClobTransformer is not
> working. Please assist.
>
> Regards
> Kamal
>

Clob transformer not working in DIH

[Kamal Kishore Aggarwal]

Hi,

I am using solr 5.4.1. Here I am using dataimport handler to index data
with SQL Server.

I am using CLOB transformer to convert clob value to string. Indexing is
working fine but clob transformation is not working. Expected string value
is not coming for clob column. There is no error or exception coming in log.

Here is the configuration:


















I tried using RegexTransformer, it worked. But ClobTransformer is not
working. Please assist.

Regards
Kamal

Re: Blank/Null value search in term filter

[Kamal Kishore Aggarwal]

Thanks Ahmet for your response and nice suggestion.

But, I was looking if there any way out without making any configuration
change.

Please suggest.

On 02-Sep-2016 9:37 PM, "Ahmet Arslan" <iori...@yahoo.com> wrote:

>
>
> Hi Kishore,
>
> You can employ an impossible token value (say XX) for null values.
> This can be done via default value update processor factory.
> You index some placeholder token for null values.
> fq={!terms f='queryField' separator='|'}A|XX would fetche docs with A or
> null values.
> Ahmet
>
> On Friday, September 2, 2016 2:03 PM, Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
>
>
> Hi,
>
> We are using solr 5.4.1.
>
> We are using term filter for multiple value matching purpose.
> Example: fq={!terms f='queryField' separator='|'}A|B
>
> A, B, C are the possible field values for solr field "queryField". There
> can docs with null values for the same field. Now, how can I create a term
> filter in above fashion that fetches docs with A or null values.
>
> Please suggest.
>
> Regards
> Kamal
>

Blank/Null value search in term filter

[Kamal Kishore Aggarwal]

Hi,

We are using solr 5.4.1.

We are using term filter for multiple value matching purpose.
Example: fq={!terms f='queryField' separator='|'}A|B

A, B, C are the possible field values for solr field "queryField". There
can docs with null values for the same field. Now, how can I create a term
filter in above fashion that fetches docs with A or null values.

Please suggest.

Regards
Kamal

Re: Find records with no values in solr.LatLongType fied type

[Kamal Kishore Aggarwal]

Thanks Erick..it worked..

On Wed, Sep 16, 2015 at 9:21 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> Top level queries need a *:* in front, something like
> q=*:* -usrlatlong_0_coordinate:[* TO *]
>
> I just took a quick check and just using usrlatlong:[* TO *]
> encounters a parse error.
>
> P.S. It would help if you told us what you _did_ receive
> when you tried your options. Parse errors? All docs?
>
> Best,
> Erick
>
> On Mon, Sep 14, 2015 at 10:58 PM, Kamal Kishore Aggarwal
> <kkroyal@gmail.com> wrote:
> > Hi,
> >
> > I am working on solr 4.8,1. I am trying to find the docs where
> latlongtype
> > have null values.
> >
> > I have tried using these, but not getting the results :
> >
> > 1) http://localhost:8984/solr/IM-Search/select?q.alt=-usrlatlong:[' '
> TO *]
> >
> > 2) http://localhost:8984/solr/IM-Search/select?q.alt=-usrlatlong:[* TO
> *]
> >
> > Here's the configurations :
> >>  >> subFieldSuffix="_coordinate"/>
> >>  >> required="false" multiValued="false" />
> >
> >
> > Please help.
>

Find records with no values in solr.LatLongType fied type

[Kamal Kishore Aggarwal]

Hi,

I am working on solr 4.8,1. I am trying to find the docs where latlongtype
have null values.

I have tried using these, but not getting the results :

1) http://localhost:8984/solr/IM-Search/select?q.alt=-usrlatlong:[' ' TO *]

2) http://localhost:8984/solr/IM-Search/select?q.alt=-usrlatlong:[* TO *]

Here's the configurations :
>  subFieldSuffix="_coordinate"/>
>  required="false" multiValued="false" />


Please help.

Re: Solr Replication sometimes coming in log files

[Kamal Kishore Aggarwal]

Can anybody suggest me something..

On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi Team,
>
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
>
>
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1 is in another server in different zone (say zone B). There is latency of
> 40 ms between two zones.
>
> Now, a days we are facing high load on Slave 1 & we suspect that it is due
> to delay in data replication from Master server. These days we are finding
> these below mentioned replication information in log files, but such lines
> are not in previous files on the Slave 1 server. Also, such information is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
>
>
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708786003} status=0 QTime=173
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708787976} status=0 QTime=1807
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708791563} status=0 QTime=7140
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708800450} status=0 QTime=1679
>
>
>
> Please confirm if we our thought that increased replication time (which
> can be due to servers connectivity issues) is the reason for high load on
> solr.
>
> Regards
> Kamal Kishore
>
>

Solr Replication sometimes coming in log files

[Kamal Kishore Aggarwal]

Hi Team,

I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
configuration has master & slave ( 2 Slaves) architecture.


Master & Slave 2 are in same server location (say zone A) , whereas Slave 1
is in another server in different zone (say zone B). There is latency of 40
ms between two zones.

Now, a days we are facing high load on Slave 1 & we suspect that it is due
to delay in data replication from Master server. These days we are finding
these below mentioned replication information in log files, but such lines
are not in previous files on the Slave 1 server. Also, such information is
not there in any Slave 2 log files (might be due to same zone of master &
slave 2).


> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708786003} status=0 QTime=173
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708787976} status=0 QTime=1807
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708791563} status=0 QTime=7140
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708800450} status=0 QTime=1679



Please confirm if we our thought that increased replication time (which can
be due to servers connectivity issues) is the reason for high load on solr.

Regards
Kamal Kishore

Re: Protwords in solr spellchecker

[Kamal Kishore Aggarwal]

Hi David,

This one is a good suggestion. But, if add these *adult* keywords in the
stopwords.txt file, it will be requiring the re-indexing of these keywords
related data.

How can I see the change instantly. Is there any other great suggestion
that you can suggest me.




On Thu, Jul 9, 2015 at 12:09 PM, davidphilip cherian 
davidphilipcher...@gmail.com wrote:

 The best bet is to use solr.StopFilterFactory.
 Have all such words added to stopwords.txt and add this filter to your
 analyzer.

 Reference links

 https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.StopFilterFactory

 https://cwiki.apache.org/confluence/display/solr/Filter+Descriptions#FilterDescriptions-StopFilter

 HTH


 On Thu, Jul 9, 2015 at 11:50 AM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

  Hi Team,
 
  I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. Is there
  any feature by which I can refrain the following words to appear in spell
  suggestion.
 
  For example: Somebody searches for sexe, I does not want to show him sex
 as
  the spell suggestion via solr. How can I stop these type of keywords to
 be
  shown in suggestion.
 
  Any help is appreciated.
 
 
  Regards
  Kamal Kishore
  Solr Beginner
 

Protwords in solr spellchecker

[Kamal Kishore Aggarwal]

Hi Team,

I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. Is there
any feature by which I can refrain the following words to appear in spell
suggestion.

For example: Somebody searches for sexe, I does not want to show him sex as
the spell suggestion via solr. How can I stop these type of keywords to be
shown in suggestion.

Any help is appreciated.


Regards
Kamal Kishore
Solr Beginner

Re: Java upgrade for solr in master-slave configuration

[Kamal Kishore Aggarwal]

Hi,

Anybody tried upgrading master first prior to slave Java upgrade. Please
suggest.




On Tue, May 19, 2015 at 6:50 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 5/19/2015 12:21 AM, Kamal Kishore Aggarwal wrote:
  I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
  configuration has slave  master architecture. I am looking forward to
  upgrade Java from 1.7 to 1.8 version in order to take advantage of memory
  optimization done in latest version.
 
  So, I am confused if I should upgrade java first on master server and
 then
  on slave server or the other way round. What should be the ideal steps,
 so
  that existing solr index and other things should not get corrupted .
 Please
  suggest.

 I am not aware of any changes in index format resulting from changing
 your Java version.  It should not matter which machines you upgrade first.

 Thanks,
 Shawn

Java upgrade for solr in master-slave configuration

[Kamal Kishore Aggarwal]

Hi,

I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
configuration has slave  master architecture. I am looking forward to
upgrade Java from 1.7 to 1.8 version in order to take advantage of memory
optimization done in latest version.

So, I am confused if I should upgrade java first on master server and then
on slave server or the other way round. What should be the ideal steps, so
that existing solr index and other things should not get corrupted . Please
suggest.


Regards
Kamal

Solr Performance with Ram size variation

[Kamal Kishore Aggarwal]

Hi,

As per this article, the linux machine is preferred to have 1.5 times RAM
with respect to index size. So, to verify this, I tried testing the solr
performance in different volumes of RAM allocation keeping other
configuration (i.e Solid State Drives, 8 core processor, 64-Bit) to be same
in both the cases. I am using solr 4.8.1 with tomcat server.

https://wiki.apache.org/solr/SolrPerformanceProblems

1) Initially, the linux machine had 32 GB RAM, out of which I allocated
14GB to solr.

export CATALINA_OPTS=-Xms2048m -Xmx14336m -XX:+UseConcMarkSweepGC
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -Xloggc:./logs/info_error/tomcat_gcdetails.log

The average search time for 1000 queries 300ms.

2) After that, RAM was increased to 68 GB, out of which I allocated 40GB to
Solr. Now, on a strange note, the average search time for the same set of
queries was 3000ms.

Now, after this, I reduced solr allocated RAM to 25GB on 68GB machine. But,
still the search time was higher as compared to first case.

What am I missing. Please suggest.

Re: External File Field eating memory

[Kamal Kishore Aggarwal]

Hi Apporva,

This was my master server replication configuration:

core/conf/solrconfig.xml

requestHandler name=/replication class=solr.ReplicationHandler 
 lst name=master
 str name=replicateAftercommit/str
 str name=replicateAfterstartup/str
 str name=confFiles../data/external_eff_views/str
 /lst
 /requestHandler


It is only configuration files that can be replicated. So, when I wrote the
above config. The external files was getting replicated in
core/conf/data/external_eff_views.
But for solr to read the external file, it looks for it into
core/data/external_eff_views
location. So firstly the file was not getting replicated properly.
Therefore, I did not opted the option of replicating the eff file.

And the second thing is that whenever there is a change in configuration
files, the core gets reloaded by itself to reflect the changes. I am not
sure if you can disable this reloading.

Finally, I thought of creating files on slaves in a different way.

Thanks
Kamal


On Tue, Jul 15, 2014 at 11:00 AM, Apoorva Gaurav apoorva.gau...@myntra.com
wrote:

 Hey Kamal,
 What all config changes have you done to establish replication of external
 files and how have you disabled role reloading?


 On Wed, Jul 9, 2014 at 11:30 AM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

  Hi All,
 
  It was found that external file, which was getting replicated after every
  10 minutes was reloading the core as well. This was increasing the query
  time.
 
  Thanks
  Kamal Kishore
 
 
 
  On Thu, Jul 3, 2014 at 12:48 PM, Kamal Kishore Aggarwal 
  kkroyal@gmail.com wrote:
 
   With the above replication configuration, the eff file is getting
   replicated at core/conf/data/external_eff_views (new dir data is being
   created in conf dir) location, but it is not getting replicated at
  core/data/external_eff_views
   on slave.
  
   Please help.
  
  
   On Thu, Jul 3, 2014 at 12:21 PM, Kamal Kishore Aggarwal 
   kkroyal@gmail.com wrote:
  
   Thanks for your guidance Alexandre Rafalovitch.
  
   I am looking into this seriously.
  
   Another question is that I facing error in replication of eff file
  
   This is master replication configuration:
  
   core/conf/solrconfig.xml
  
   requestHandler name=/replication class=solr.ReplicationHandler 
   lst name=master
   str name=replicateAftercommit/str
   str name=replicateAfterstartup/str
   str name=confFiles../data/external_eff_views/str
   /lst
   /requestHandler
  
  
   The eff file is present at core/data/external_eff_views location.
  
  
   On Thu, Jul 3, 2014 at 11:50 AM, Shalin Shekhar Mangar 
   shalinman...@gmail.com wrote:
  
   This might be related:
  
   https://issues.apache.org/jira/browse/SOLR-3514
  
  
   On Sat, Jun 28, 2014 at 5:34 PM, Kamal Kishore Aggarwal 
   kkroyal@gmail.com wrote:
  
Hi Team,
   
I have recently implemented EFF in solr. There are about 1.5
   lacs(unsorted)
values in the external file. After this implementation, the server
  has
become slow. The solr query time has also increased.
   
Can anybody confirm me if these issues are because of this
   implementation.
Is that memory does EFF eats up?
   
Regards
Kamal Kishore
   
  
  
  
   --
   Regards,
   Shalin Shekhar Mangar.
  
  
  
  
 



 --
 Thanks  Regards,
 Apoorva

Re: Integrate NLP to solr

[Kamal Kishore Aggarwal]

Hi Ahmet,

The installation step at wiki.apache.org/solr/OpenNLP ran succesfully. I
copied the following jars:

solr/contrib/opennlp/lib/opennlp-maxent-3.0.3.jar
 solr/contrib/opennlp/lib/opennlp-tools-1.5.3.jar


in the lib directory of the solr.

I made the changes in the schema.xml. Created opennlp dir in conf/
directory of the core  add en-* files in it.

Still I am getting the following error:


 Caused by: java.lang.ClassNotFoundException: solr.OpenNLPTokenizerFactory
 at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
 java.net.URLClassLoader$1.run(URLClassLoader.java:355)


Please help.


Regrads
Kamal Kishore



On Thu, Jul 10, 2014 at 10:01 PM, Ahmet Arslan iori...@yahoo.com.invalid
wrote:

 Hi Kamal,

 Uncommitted code may not work.

 Have you considered using
 https://cwiki.apache.org/confluence/display/solr/UIMA+Integration


 Ahmet


 On Thursday, July 10, 2014 10:33 AM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:



 Dear Team,

 I am working on solr 4.8.1 version. I want to integrate solr with NLP in
 order to improve the search relevancy. I am following the tutorials:

 https://wiki.apache.org/solr/OpenNLP
 https://issues.apache.org/jira/browse/LUCENE-2899

 Its says:

- pull the latest trunk or 4.0 branch
-

apply the latest LUCENE-2899
https://issues.apache.org/jira/browse/LUCENE-2899 patch
- do 'ant compile'
- cd solr/contrib/opennlp/src/test-files/training
- run 'bin/trainall.sh'

 What I have done is:

 - Downloaded solr-4.8.1 in /data/ - cd /data/solr-4.8.1/; - Downloaded
 all four files from issues.apache.org/jira/browse/LUCENE-2899; - patch
 -p0
 -i LUCENE-2899.patch Getting the following error: can't find file to patch
 at input line 5 Perhaps you used the wrong -p or --strip option?

 I am failing :(.   Please help.


 Regards
 Kamal Kishore

Integrate NLP to solr

[Kamal Kishore Aggarwal]

Dear Team,

I am working on solr 4.8.1 version. I want to integrate solr with NLP in
order to improve the search relevancy. I am following the tutorials:

https://wiki.apache.org/solr/OpenNLP
https://issues.apache.org/jira/browse/LUCENE-2899

Its says:

   - pull the latest trunk or 4.0 branch
   -

   apply the latest LUCENE-2899
   https://issues.apache.org/jira/browse/LUCENE-2899 patch
   - do 'ant compile'
   - cd solr/contrib/opennlp/src/test-files/training
   - run 'bin/trainall.sh'

What I have done is:

- Downloaded solr-4.8.1 in /data/ - cd /data/solr-4.8.1/; - Downloaded
all four files from issues.apache.org/jira/browse/LUCENE-2899; - patch -p0
-i LUCENE-2899.patch Getting the following error: can't find file to patch
at input line 5 Perhaps you used the wrong -p or --strip option?

I am failing :(.   Please help.


Regards
Kamal Kishore

Re: External File Field eating memory

[Kamal Kishore Aggarwal]

Hi All,

It was found that external file, which was getting replicated after every
10 minutes was reloading the core as well. This was increasing the query
time.

Thanks
Kamal Kishore



On Thu, Jul 3, 2014 at 12:48 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 With the above replication configuration, the eff file is getting
 replicated at core/conf/data/external_eff_views (new dir data is being
 created in conf dir) location, but it is not getting replicated at 
 core/data/external_eff_views
 on slave.

 Please help.


 On Thu, Jul 3, 2014 at 12:21 PM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

 Thanks for your guidance Alexandre Rafalovitch.

 I am looking into this seriously.

 Another question is that I facing error in replication of eff file

 This is master replication configuration:

 core/conf/solrconfig.xml

 requestHandler name=/replication class=solr.ReplicationHandler 
 lst name=master
 str name=replicateAftercommit/str
 str name=replicateAfterstartup/str
 str name=confFiles../data/external_eff_views/str
 /lst
 /requestHandler


 The eff file is present at core/data/external_eff_views location.


 On Thu, Jul 3, 2014 at 11:50 AM, Shalin Shekhar Mangar 
 shalinman...@gmail.com wrote:

 This might be related:

 https://issues.apache.org/jira/browse/SOLR-3514


 On Sat, Jun 28, 2014 at 5:34 PM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

  Hi Team,
 
  I have recently implemented EFF in solr. There are about 1.5
 lacs(unsorted)
  values in the external file. After this implementation, the server has
  become slow. The solr query time has also increased.
 
  Can anybody confirm me if these issues are because of this
 implementation.
  Is that memory does EFF eats up?
 
  Regards
  Kamal Kishore
 



 --
 Regards,
 Shalin Shekhar Mangar.

Error in replication eff files

[Kamal Kishore Aggarwal]

Dear Team,


I am working on external file field. But, I does not know the configuration
of how to replicate the EFF files.

This is master replication configuration:

core/conf/solrconfig.xml

requestHandler name=/replication class=solr.ReplicationHandler 
lst name=master
str name=replicateAftercommit/str
str name=replicateAfterstartup/str
str name=confFiles../data/external_eff_views/str
/lst
/requestHandler

The eff file is present at core/data/external_eff_views location.

Thanks  Regards
Kamal Kishore

Re: External File Field eating memory

[Kamal Kishore Aggarwal]

Thanks for your guidance Alexandre Rafalovitch.

I am looking into this seriously.

Another question is that I facing error in replication of eff file

This is master replication configuration:

core/conf/solrconfig.xml

requestHandler name=/replication class=solr.ReplicationHandler 
 lst name=master
 str name=replicateAftercommit/str
 str name=replicateAfterstartup/str
 str name=confFiles../data/external_eff_views/str
 /lst
 /requestHandler


The eff file is present at core/data/external_eff_views location.


On Thu, Jul 3, 2014 at 11:50 AM, Shalin Shekhar Mangar 
shalinman...@gmail.com wrote:

 This might be related:

 https://issues.apache.org/jira/browse/SOLR-3514


 On Sat, Jun 28, 2014 at 5:34 PM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

  Hi Team,
 
  I have recently implemented EFF in solr. There are about 1.5
 lacs(unsorted)
  values in the external file. After this implementation, the server has
  become slow. The solr query time has also increased.
 
  Can anybody confirm me if these issues are because of this
 implementation.
  Is that memory does EFF eats up?
 
  Regards
  Kamal Kishore
 



 --
 Regards,
 Shalin Shekhar Mangar.

Re: External File Field eating memory

[Kamal Kishore Aggarwal]

With the above replication configuration, the eff file is getting
replicated at core/conf/data/external_eff_views (new dir data is being
created in conf dir) location, but it is not getting replicated at
core/data/external_eff_views
on slave.

Please help.


On Thu, Jul 3, 2014 at 12:21 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 Thanks for your guidance Alexandre Rafalovitch.

 I am looking into this seriously.

 Another question is that I facing error in replication of eff file

 This is master replication configuration:

 core/conf/solrconfig.xml

 requestHandler name=/replication class=solr.ReplicationHandler 
 lst name=master
 str name=replicateAftercommit/str
 str name=replicateAfterstartup/str
 str name=confFiles../data/external_eff_views/str
 /lst
 /requestHandler


 The eff file is present at core/data/external_eff_views location.


 On Thu, Jul 3, 2014 at 11:50 AM, Shalin Shekhar Mangar 
 shalinman...@gmail.com wrote:

 This might be related:

 https://issues.apache.org/jira/browse/SOLR-3514


 On Sat, Jun 28, 2014 at 5:34 PM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

  Hi Team,
 
  I have recently implemented EFF in solr. There are about 1.5
 lacs(unsorted)
  values in the external file. After this implementation, the server has
  become slow. The solr query time has also increased.
 
  Can anybody confirm me if these issues are because of this
 implementation.
  Is that memory does EFF eats up?
 
  Regards
  Kamal Kishore
 



 --
 Regards,
 Shalin Shekhar Mangar.

Re: External File Field eating memory

[Kamal Kishore Aggarwal]

Any replies ??


On Sat, Jun 28, 2014 at 5:34 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 Hi Team,

 I have recently implemented EFF in solr. There are about 1.5
 lacs(unsorted) values in the external file. After this implementation, the
 server has become slow. The solr query time has also increased.

 Can anybody confirm me if these issues are because of this implementation.
 Is that memory does EFF eats up?

 Regards
 Kamal Kishore

External File Field eating memory

[Kamal Kishore Aggarwal]

Hi Team,

I have recently implemented EFF in solr. There are about 1.5 lacs(unsorted)
values in the external file. After this implementation, the server has
become slow. The solr query time has also increased.

Can anybody confirm me if these issues are because of this implementation.
Is that memory does EFF eats up?

Regards
Kamal Kishore

Nested boosting in map function in solr?

[Kamal Kishore Aggarwal]

Dear Team,

I am trying to implement nested boosting in solr using map function.

http://www.example.com:8984/solr/collection1/select?q=laundry
servicesboost=map(query({!dismax
qf=titlex !v=$ql3 pf=}),0,0,1,map(query({!dismax qf=city v='mumbai'
pf=}),0,0,1,15))ql3=laundry services.

But this is reporting the following error:

msg:org.apache.solr.search.SyntaxError: Expected ',' at position 69 in
'map(query({!dismax qf=titlex v=$ql3 pf=\\}),0,0,1,map(query({!dismax
qf=city v='\mumbai\' pf=\\}),0,0,1,15))',


Regards

Kamal Kishore

Re: Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

Hi,

I have changed the code. But now it is showing the following errors:

Caused by: java.lang.NoSuchMethodException:
org.apache.lucene.analysis.ExtendedNameFilterFactory.init(java.util.Map)
Here's the new java code : http://pastebin.com/J8q4JLgP

A urgent help is appreciated. :)

Thanks
Kamal


On Wed, May 21, 2014 at 1:38 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 Thanks Shawn for quick reply.

 I am trying to change the code (removing the errors from the code shown in
 image)  will test the filter after that  will update here.

 Thanks
 Kamal Kishore


 On Mon, May 19, 2014 at 10:17 PM, Shawn Heisey s...@elyograg.org wrote:

 On 5/19/2014 1:10 AM, Kamal Kishore Aggarwal wrote:
  I have created a custom filter factory for solr 4.2. It is working good.
  But when I am trying to upgarde solr-4.2 to 4.7 version, it is reporting
  errors.
 
  Caused by: org.apache.solr.common.SolrException: Plugin init failure for
  [schema.xml] analyzer/filter: Error instantiating class:
  'org.apache.lucene.analysis.ExtendedNameFilterFactory'
 
  Here's the java code :http://pastebin.com/REu6cJxR,
  http://pastebin.com/380YZaAM
 
  It was working good with solr 4.2. Can anybody tell me the changes that
 I
  need to make for running it in solr 4.7.1?

 I probably would have used something like com.company.lucene.filter as
 the package name, but what you name the package is up to you, and will
 have no effect on your problem.

 I dropped your code into a trunk checkout (the code that will eventually
 become Lucene/Solr 5.0).  The Eclipse editor looked like this for your
 factory class:

 https://www.dropbox.com/s/tobjrk0riq7vqqt/extendednamefilterfactory.png

 The errors shown for the three red marks are:

 * Implicit super constructor TokenFilterFactory() is undefined for
 default constructor. Must define an explicit constructor.
 * The method init(MapString,String) is undefined for the type
 TokenFilterFactory.
 * The method getInt(MapString,String, String, int) in the type
 AbstractAnalysisFactory is not applicable for the arguments (String, int)

 I don't have easy access to a 4.x checkout right this minute, but the
 errors there are probably similar.

 Your actual filter class was all good except for three eclipse warnings
 that are fixed by adding a type argument of String to the code.

 Thanks,
 Shawn

Re: Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

Even after making the same java version, it is not working. I am using

java.​runtime.​version:1.7.0_55-b13








On Tue, Jun 3, 2014 at 2:05 PM, rulinma ruli...@gmail.com wrote:

 normal, rewrite filter.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Custom-filter-not-working-with-solr-4-7-1-tp4136824p4139506.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

Thanks Shawn for quick reply.

I am trying to change the code (removing the errors from the code shown in
image)  will test the filter after that  will update here.

Thanks
Kamal Kishore


On Mon, May 19, 2014 at 10:17 PM, Shawn Heisey s...@elyograg.org wrote:

 On 5/19/2014 1:10 AM, Kamal Kishore Aggarwal wrote:
  I have created a custom filter factory for solr 4.2. It is working good.
  But when I am trying to upgarde solr-4.2 to 4.7 version, it is reporting
  errors.
 
  Caused by: org.apache.solr.common.SolrException: Plugin init failure for
  [schema.xml] analyzer/filter: Error instantiating class:
  'org.apache.lucene.analysis.ExtendedNameFilterFactory'
 
  Here's the java code :http://pastebin.com/REu6cJxR,
  http://pastebin.com/380YZaAM
 
  It was working good with solr 4.2. Can anybody tell me the changes that I
  need to make for running it in solr 4.7.1?

 I probably would have used something like com.company.lucene.filter as
 the package name, but what you name the package is up to you, and will
 have no effect on your problem.

 I dropped your code into a trunk checkout (the code that will eventually
 become Lucene/Solr 5.0).  The Eclipse editor looked like this for your
 factory class:

 https://www.dropbox.com/s/tobjrk0riq7vqqt/extendednamefilterfactory.png

 The errors shown for the three red marks are:

 * Implicit super constructor TokenFilterFactory() is undefined for
 default constructor. Must define an explicit constructor.
 * The method init(MapString,String) is undefined for the type
 TokenFilterFactory.
 * The method getInt(MapString,String, String, int) in the type
 AbstractAnalysisFactory is not applicable for the arguments (String, int)

 I don't have easy access to a 4.x checkout right this minute, but the
 errors there are probably similar.

 Your actual filter class was all good except for three eclipse warnings
 that are fixed by adding a type argument of String to the code.

 Thanks,
 Shawn

NumberFormatException in solr.SpatialRecursivePrefixTreeFieldType in solr upgrade

[Kamal Kishore Aggarwal]

I am using following field type with solr 4.2  its working fine.

field name=latlong type=location_rpt indexed=true stored=true
required=false multiValued=false /

fieldType name=location_rpt
class=solr.SpatialRecursivePrefixTreeFieldType
   distErrPct=0.025
   maxDistErr=0.09
   units=degrees
/
But, when I am upgrading solr to solr 4.7.1, it is reporting following
errors while posting docs:
Caused by: com.spatial4j.core.exception.InvalidShapeException:
java.lang.NumberFormatException: For input string: 78.42968,30.7,78.45
at
org.apache.solr.util.SpatialUtils.parsePoint(SpatialUtils.java:68)
at
org.apache.solr.util.SpatialUtils.parsePointSolrException(SpatialUtils.java:84)
... 45 more
Caused by: java.lang.NumberFormatException: For input string:
78.42968,30.7,78.45

I am posting following doc value to solr.
   field name='latlong'![CDATA[30.9308,78.42968,30.7,78.45]]/field

With Thanks  Regards
Kamal Kishore

Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

Dear Team,

I have created a custom filter factory for solr 4.2. It is working good.
But when I am trying to upgarde solr-4.2 to 4.7 version, it is reporting
errors.

Caused by: org.apache.solr.common.SolrException: Plugin init failure for
[schema.xml] analyzer/filter: Error instantiating class:
'org.apache.lucene.analysis.ExtendedNameFilterFactory'

Here's the java code :http://pastebin.com/REu6cJxR,
http://pastebin.com/380YZaAM

It was working good with solr 4.2. Can anybody tell me the changes that I
need to make for running it in solr 4.7.1?


With Regards  Thanks

Kamal Kishore

Subscribe for Solr Mails

[Kamal Kishore Aggarwal]

Dear Team,


I am new to solr. Please subscribe me for solr mails.


With Regards  Thanks
Kamal Kishore

Re: Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

I am new to solr. I am not getting your answer. Can you please explain in
more detail.


On Mon, May 19, 2014 at 12:40 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 Dear Team,

 I have created a custom filter factory for solr 4.2. It is working good.
 But when I am trying to upgarde solr-4.2 to 4.7 version, it is reporting
 errors.

 Caused by: org.apache.solr.common.SolrException: Plugin init failure for
 [schema.xml] analyzer/filter: Error instantiating class:
 'org.apache.lucene.analysis.ExtendedNameFilterFactory'

 Here's the java code :http://pastebin.com/REu6cJxR,
 http://pastebin.com/380YZaAM

 It was working good with solr 4.2. Can anybody tell me the changes that I
 need to make for running it in solr 4.7.1?


 With Regards  Thanks

 Kamal Kishore

Re: Custom filter not working with solr 4.7.1

[Kamal Kishore Aggarwal]

I am new to solr. I am not getting your answer. Can you please explain in
more detail.


On Mon, May 19, 2014 at 4:02 PM, Kamal Kishore Aggarwal 
kkroyal@gmail.com wrote:

 I am new to solr. I am not getting your answer. Can you please explain in
 more detail.


 On Mon, May 19, 2014 at 12:40 PM, Kamal Kishore Aggarwal 
 kkroyal@gmail.com wrote:

 Dear Team,

 I have created a custom filter factory for solr 4.2. It is working good.
 But when I am trying to upgarde solr-4.2 to 4.7 version, it is reporting
 errors.

 Caused by: org.apache.solr.common.SolrException: Plugin init failure for
 [schema.xml] analyzer/filter: Error instantiating class:
 'org.apache.lucene.analysis.ExtendedNameFilterFactory'

 Here's the java code :http://pastebin.com/REu6cJxR,
 http://pastebin.com/380YZaAM

 It was working good with solr 4.2. Can anybody tell me the changes that I
 need to make for running it in solr 4.7.1?


 With Regards  Thanks

 Kamal Kishore