Re: Solr Delta Import Issue

[~$alpha`]

Time for import - 5-6 minutes
Warmup time - 40seconds

autoCommit and autoSoftCommit setting both disabled and We fire commit only
after the import is completed.

I have some more doubt 
1.  In case of master-slave, is auto warm strategy is available for slaves 

2. Also should I also have a limit on max connection to make server Solr
server won't go down and if yes how to choose that max no. ? 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr Delta Import Issue

[~$alpha`]

 

SOLR VERSION 6.0.0

As seen in the image, There is a spike which can be observed at every 2
hours.
i.e whenever delta import runs

 1. Response time doubles
 2. CPU load average doubles and if it runs to near to peal hour than it
goes even to 10 times.
 

**My Cache Settings are**

 1. 



 

  

true

20
200
false

Most of my queries are CPU centric as they involved lots of IN queries and
NOT IN Queries. Also, have if cond for scoring. Assuming my queries will
continue to be CPU centric.

**Help**
What Am I doing wrong as my delta import causing so much high response?
Index size : 2GB
Servers: 4
Serving: 1Lac per hour
Also, Delta update results in 2lac record update out of 10lac as one of the
fields of solr (last login) changed frequently.
  






--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: CustomScoreProvider Sucks, Need Help

[~$alpha`]

But why when i am using customize sort, with just return it should work fast



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: SolrException: Error Instantiating queryParser, com.site.s.CustomQParserPlugin failed to instantiate org.apache.solr.search.QParserPlugin

[~$alpha`]

Resolved by inclding solr on depency using maven



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

CustomScoreProvider Sucks, Need Help

[~$alpha`]

public class MatchingScoreProvider extends CustomScoreProvider {

}

Issues:
1. CustomScoreProvider works but is too slow.
Even when I am writing return 1 on 1st line it's still taking  5 seconds for
6lac row.

2. I added  @Slf4j but still not logger not working



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Custom Solr Sorting + Query

[~$alpha`]

I want to write a custom Solr query and sorting and when I am calling
http://127.0.0.1:8983/solr/techproducts/select?q={!myparsername}12=uid
desc, some of the section as marked is not getting called.

Issue: Documentation of solr customization is pretty poor

Help: Can you please tell why my code is not getting called.

Why I needed customization: Based on logged in user info like user age,
location and 10 other parameters I need to sort. Also, need sorting value
for tagging.
Default Solr expression is very costly.

=== class-1==

public class MyPlugin extends QParserPlugin {

@Override
public QParser createParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
return new CustomQParser(qstr, localParams, params, req);   

}

private static class CustomQParser extends QParser{

private Query innerquery;

public CustomQParser(String qstr, SolrParams localParams, 
SolrParams
params, SolrQueryRequest req) {
//--> getting called ...(1)
super(qstr, localParams, params, req);
try {
QParser parser = getParser(qstr, getReq());
this.innerquery = parser.parse();
}catch(SyntaxError ex) {
throw new RuntimeException("error parsing 
query", ex);
}
}

@Override
public Query parse() throws SyntaxError {
//--> getting called ...(2)
return new MatchTheQuery(innerquery);
}

}

}


== class-2 == 
public class MatchTheQuery extends CustomScoreQuery {

private Query subquery;

public MatchTheQuery(Query subquery) {
//--> getting called ...(3)
super(subquery); 
this.subquery = subquery;
}

@Override
protected CustomScoreProvider getCustomScoreProvider(LeafReaderContext
context) {
// not getting called :-(
return new MyScoreProvider(subquery, context);
}
}

== class-3 == 
public class MyScoreProvider extends CustomScoreProvider {

private Query subquery;

public MyScoreProvider(Query subquery, LeafReaderContext context) {
super(context);
this.subquery = subquery;
}

@Override
public float customScore(int doc, float subqueryScore, float
valSrcScores[]) throws IOException {

// not getting called :-(

}


}



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SolrException: Error Instantiating queryParser, com.site.s.CustomQParserPlugin failed to instantiate org.apache.solr.search.QParserPlugin

[~$alpha`]

> I have created a custom query parser for Solr
public class CustomQParserPlugin extends QParserPlugin {
}

>Then I have imported the jar into Solr lib directory. i.e placed in
server/lib directory

>I restarted the Solr

>  getting below error
SolrException: Error Instantiating queryParser,
com.site.s.CustomQParserPlugin failed to instantiate
org.apache.solr.search.QParserPlugin

I am getting the above error which means I am able to include my jar but no
idea why I am getting error




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr Plugins Documentation

[~$alpha`]

I got a basic understanding of how we can create the custom Solr plugin as
illustrated in link
https://dzone.com/articles/create-custom-solr-queryparser

But I don't know the next step as for how can we create a jar.
like do we need to import Solr source code ??

It would be of great help if someone can share the complete step to how to
create custom sort 

Note : I have basic understanding of java and surely can write java code 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

cpu utlization is already on higher end, so VM wont seems a solution



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Expression Slow

[~$alpha`]

It cannot be done at indexing time. its based onloggedin user info



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

top command in linux 



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

Yes, but how to move ahead  now.
Its strange solr4 is better behaving than solr6



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr Expression Slow

[~$alpha`]

In the below Solr query I am sorting based on the below expression.

> sum(if(and(tf(CRITERIA1_FILTER,Y),if(tf(PARTNER_CRITERIA1,N),0,1)),1,0),if(and(tf(CRITERIA2_FILTER,Y),if(tf(PARTNER_CRITERIA2,1),0,1)),1,0))
> asc

I have mentioned PARTNER_CRITERIA1 and PARTNER_CRITERIA2 in the expression.
I have similar10 expression like this.

I want to remove expression as it is slow,
Is it better to use custom sorting for such case and if yes, can someone
help how to do so?





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

I cant test on more as performance is already degraded.
Its a 32core system and load 25 means 2500% cpu



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

Hits 41 : 
Avg response time470ms vs 380ms 
CPU Load  reaches6 vs 10 

Hits 82: 
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours) 
CPU Load  reaches11 vs 25



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

When both solr4 and solr6 have concurrent hits:
1. 30 to 40 : 
Avg response time 470ms vs 380ms
Load 6 vs 10

1. 80 to 90 : 
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
Load 11 vs 25




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

Other resource is not using the cpu.
Its true that response is better to 300ms from 350ms but cpu usage almost
doubled?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr4 To Solr6 CPU load issues

[~$alpha`]

Config : 64GB RAM 32 CORE CPU
but i have given 20Gb to solr JVM.. Also its a shared resource



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr4 To Solr6 CPU load issues

[~$alpha`]

I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
overall the CPU load on solr6.6 is double the load on solr4.0 and hence I am
not able to roll solr6.6 to 100% of my traffic.

*Some Key Stats In Performance of Sol6 Vs Solr4*
Document cache usage increased from .98 from .14
Query Result cache usage increased from .10 from .24
Filter cache same as .94
Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
multivalued concept was changed from solr4 to solr6)

*Please Help Note: I have given document cache 3 times memory for
doc.cache.*





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Solr4 To Solr5 Upgrade Causing CPU issues

[~$alpha`]

Hi 

Recently upgraded from solr4.0(tomcat) to solr5.3(tomcat) and my performance
degraded to a very high level. 

Issues: My CPU consumption increased to the very high level of the same
traffic. 
Note: My queries contains lots of in queries.

What can be the issue here.
I also tried solr6 with jetty and still the same performance, Please help.

 





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr4-To-Solr5-Upgrade-Causing-CPU-issues-tp4328155.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Update If Record Exists ?

[~$alpha`]

My version is timestamp.
1561379927065886720

So can i use it ?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Update-If-Record-Exists-tp4323767p4324152.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr Update If Record Exists ?

[~$alpha`]

*SOLR_URL/update -d \'
[
 {"id" : "1",
  "ONLINE" : {"set":"1"}
 }
]'*

I am using solr6.3. Above command works fine as it updates online flag to 1
for id=1. But the issue is if the record is not present then it adds a value
as id=1 and online=1 which is not desired.

So question is, is it possible that so updates the value only if the record
is present in the solr.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Update-If-Record-Exists-tp4323767.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Sorl 6 with jetty issues

[~$alpha`]

Face similar issue on solr 6.1.0



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Sorl-6-with-jetty-issues-tp4321291p4321308.html
Sent from the Solr - User mailing list archive at Nabble.com.

Sorl 6 with jetty issues

[~$alpha`]

Issues with solr settings while migrating from solr 4.0 to solr6.0.

Issue Faced: My CPU consumption goes to unacceptable levels. ie. load on
solr4.0 is between 6 to 10 while the load on solr 6 reaches 100 and since
its the production i rolled back quickly.

My Solr4 setting

 - Running on tomcat
 - JVM Memory : 16GB
 - 24 core cpu
 - JVM settings :
   - JVM Runtime Java HotSpot(TM) 64-Bit Server VM (24.45-b08) 
   - Processors   24 
   - Args : Paths mentioned here


**My Solr6 setting**

 - Running on jetty
 - JVM Memory : 20GB
 - 32 core cpu
 - JVM settings :
   - Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_45 25.45-b02
   - Processors   32
   - Args
  - DSTOP.KEY=solrrocks
  - DSTOP.PORT=7983
  - Djetty.home=/usr/local/solr-6.4.1/server-Djetty.port=8983
  -
Dlog4j.configuration=file:/usr/local/solr-6.4.1/example/resources/log4j.properties
  -
Dsolr.install.dir=/usr/local/solr-6.4.1-Dsolr.log.dir=/usr/local/solr-6.4.1/example/techproducts/solr/../logs
  - Dsolr.log.muteconsole
  -
Dsolr.solr.home=/usr/local/solr-6.4.1/example/techproducts/solr-Duser.timezone=US/Eastern
  - XX:+AggressiveOpts
  - XX:+CMSParallelRemarkEnabled
  - XX:+CMSScavengeBeforeRemark
  - XX:+ParallelRefProcEnabled
  - XX:+PrintGCApplicationStoppedTime
  - XX:+PrintGCDateStamps
  - XX:+PrintGCDetails
  - XX:+PrintGCTimeStamps
  - XX:+PrintHeapAtGC
  - XX:+PrintTenuringDistribution
  - XX:+UseCMSInitiatingOccupancyOnly
  - XX:+UseConcMarkSweepGC
  - XX:+UseGCLogFileRotation
  - XX:-UseSuperWord
  - XX:CMSFullGCsBeforeCompaction=1
  - XX:CMSInitiatingOccupancyFraction=70
  - XX:CMSMaxAbortablePrecleanTime=6000
  - XX:CMSTriggerPermRatio=80
  - XX:GCLogFileSize=20M
  - XX:MaxTenuringThreshold=8
  - XX:NewRatio=2
  - XX:NumberOfGCLogFiles=9
  - XX:OnOutOfMemoryError=/usr/local/solr-6.4.1/bin/oom_solr.sh 8983
/usr/local/solr-6.4.1/example/techproducts/solr/../logs
  - XX:PretenureSizeThreshold=64m
  - XX:SurvivorRatio=15
  -
XX:TargetSurvivorRatio=90-Xloggc:/usr/local/solr-6.4.1/example/techproducts/solr/../logs/solr_gc.log-Xms21g-Xmx21g-Xss256k-verbose:gc
What i looking for

My guess its related to gc setting of jetty as i am not expert in
jetty(java8).please help how to tune these settings. Also how should i
chosoe these values or how to to debug these issue ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Sorl-6-with-jetty-issues-tp4321291.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr6.0(Jetty) Vs Solr4.0(Tomcat)

[~$alpha`]

Hi

Recently upgraded from solr4.0(tomcat) to solr6.0(jetty) and my performance
degraded to a very high level.

Issues: My CPU consumption increased to the very high level with the same
traffic.
Note: My primary query contains lots of ids in not in query which requires
CPU.

My solr setting is the same.

Does it mean jetty performance is low as compared to tomcat when dealing
with high no. of -ve ids ?





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr6-0-Jetty-Vs-Solr4-0-Tomcat-tp4320019.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Complex Sort

[~$alpha`]

I am not sure how to use "Sort By Function" for Case.

|10#40|14#19|33#17|27#6|15#6|19#5|7#2|6#1|29#1|5#1|30#1|28#1|12#0|20#0|

Can you tell how to fetch 40 when input is 10.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Complex-Sort-tp4267155p4267165.html
Sent from the Solr - User mailing list archive at Nabble.com.

Complex Sort

[~$alpha`]

I have a column in Mysql and I need to do a complex sorting in Solr.

Considering below column

|175#40|173#17|174#13|134#11|17#8|95#4|64#3|116#3|343#0|
where 175 indicates values to be matches and 40 implies score.

So if logged in user value is 175 he will give a score of 40 and if the
value is 173 he should get a score of 17 and so on...

I have multiple such columns whose sum in above manner is my sorting
expression.

How can I do in Solr ?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Complex-Sort-tp4267155.html
Sent from the Solr - User mailing list archive at Nabble.com.

solr parallel update and total indexing Issue

[~$alpha`]

There is a bis issue in solr parallel update and total indexing 

Total Import syntax (working) 
dataimport?command=full-importcommit=trueoptimize=true 

Update syntax(working) 
solr/update?softCommit=true' -H 'Content-type:application/json' -d
'[{id:1870719,column:{set:11}}]' 


Issue: If both are run in parallel, then commit in b/w take place. 

Example: i have 10k in total indexes i fire an solr query to update 1000
records and in between i fire a total import(full indexer) what's
happening is that in between commit is taken place... i.e untill total
indexer runs i got limited records(1000). 

How to solve this ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-parallel-update-and-total-indexing-Issue-tp4132652.html
Sent from the Solr - User mailing list archive at Nabble.com.

solr parallel update and total indexing Issue

[~$alpha`]

There is a bis issue in solr parallel update and total indexing

Total Import syntax (working)
dataimport?command=full-importcommit=trueoptimize=true 

Update syntax(working)
solr/update?softCommit=true' -H 'Content-type:application/json' -d
'[{id:1870719,column:{set:11}}]'


Issue: If both are run in parallel, then commit in b/w take place.

Example: i have 10k in total indexes i fire an solr query to update 1000
records and in between i fire a total import(full indexer) what's
happening is that in between commit is taken place... i.e untill total
indexer runs i got limited records(1000).

How to solve this ?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-parallel-update-and-total-indexing-Issue-tp4131935.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to customize Solr

[~$alpha`]

the link you provided has no information about customizing 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-customize-Solr-tp4122551p4122760.html
Sent from the Solr - User mailing list archive at Nabble.com.