RE: Querying Dynamic Fields

2015-10-26 Thread Matt Kuiper (Springblox) 
Give the following a try -

http://localhost:8983/solr/core_name/admin/luke?numTerms=0 

Matt

Matt Kuiper

-Original Message-
From: Patrick Hoeffel [mailto:patrick.hoef...@issinc.com] 
Sent: Monday, October 26, 2015 4:56 PM
To: solr-user@lucene.apache.org
Subject: Querying Dynamic Fields

I have a simple Solr schema that uses dynamic fields to create most of my 
fields. This works great. Unfortunately, I now need to ask Solr to give me the 
names of the fields in the schema. I'm using:

http://localhost:8983/solr/core/schema/fields

This returns the statically defined fields, but does not return the ones that 
were created matching my dynamic definitions, such as *_s, *_i, *_txt, etc.

I know Solr is aware of these fields, because I can query against them.

What is the secret sauce to query their names and data types?

Thanks,

Patrick Hoeffel
Senior Software Engineer
Intelligent Software Solutions (www.issinc.com)
(719) 452-7371 (direct)
(719) 210-3706 (mobile)

"Bringing Knowledge to Light"

RE: Updates to index not available immediately as index scales, even with autoSoftCommit at 1 second

2014-08-12 Thread Matt Kuiper (Springblox) 
Based on your solrconfig.xml settings for the filter and queryResult caches, I 
believe Chris's initial guess is correct.  After a commit, there is likely 
plenty of time spent warming these caches due to the significantly high 
autowarm counts.


 


Suggest you try setting the autowarmcount very low or to zero, and then testing 
to confirm the  problem.

You might want to monitor if any JVM garbage collections are occurring during 
this time, and causing system pauses.  With such large caches (nominally stored 
in Old Gen) you may be setting yourself up for GCs that take a significant 
amount of time and thus add to your delay.

Matt


-Original Message-
From: cwhit [mailto:cwhi...@solinkcorp.com] 
Sent: Tuesday, August 12, 2014 11:18 AM
To: solr-user@lucene.apache.org
Subject: Re: Updates to index not available immediately as index scales, even 
with autoSoftCommit at 1 second

Immediately after triggering the update, this is what is in the logs:

/2014-08-12 12:58:48,774 | [71] | 153414367 [qtp2038499066-4772] INFO 
org.apache.solr.update.processor.LogUpdateProcessor  – [collection1] 
webapp=/solr path=/update params={wt=json} {add=[52627624 
(1476251068652322816)]} 0 34

2014-08-12 12:58:49,773 | [71] | 153415369 [commitScheduler-7-thread-1] INFO 
org.apache.solr.update.UpdateHandler  – start 
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}

2014-08-12 12:58:49,862 | [71] | 153415459 [commitScheduler-7-thread-1] INFO 
org.apache.solr.search.SolrIndexSearcher  – Opening Searcher@65c48c06 main

2014-08-12 12:58:49,874 | [71] | 153415472 [commitScheduler-7-thread-1] INFO 
org.apache.solr.update.UpdateHandler  – end_commit_flush/

The end_commit_flush leads me to believe that the soft commit has completed, 
but perhaps that thought is wrong.  There are no other logs for a while, until 


/
2014-08-12 13:03:49,556 | [71] | 153715147 [commitScheduler-6-thread-1] INFO 
org.apache.solr.update.UpdateHandler  – start 
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

2014-08-12 13:03:49,805 | [71] | 153715402 [commitScheduler-6-thread-1] INFO 
org.apache.solr.core.SolrCore  – SolrDeletionPolicy.onCommit: commits: num=2

2014-08-12 13:03:49,805 | [71] |
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@E:\Program
Files (x86)\SolrLive\SolrFiles\Solr\service\solr\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fac1a3c;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2we,generation=3758}

2014-08-12 13:03:49,805 | [71] |
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@E:\Program
Files (x86)\SolrLive\SolrFiles\Solr\service\solr\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fac1a3c;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2wf,generation=3759}

2014-08-12 13:03:49,805 | [34] | 153715403 [commitScheduler-6-thread-1] INFO 
org.apache.solr.core.SolrCore  – newest commit generation = 3759

2014-08-12 13:03:49,818 | [34] | 153715415 [commitScheduler-6-thread-1] INFO 
org.apache.solr.update.UpdateHandler  – end_commit_flush / At this point, the 
update is still not present...


/2014-08-12 13:11:45,279 | [81] | 154190876 [searcherExecutor-4-thread-1]
INFO  org.apache.solr.core.SolrCore  – QuerySenderListener sending requests
to Searcher@65c48c06 main{StandardDirectoryReader(segments_2we:82217:nrt
_qkc(4.6):C8161558/879724:delGen=275 _sra(4.6):C2943436/247953:delGen=51
_r2w(4.6):C1149753/18376:delGen=55 _rgs(4.6):C1468449/648612:delGen=107
_tdl(4.6):C583431/7873:delGen=94 _svo(4.6):C197286/7:delGen=5
_t4d(4.6):C247031/2928:delGen=36 _tkf(4.6):C111429/761:delGen=23
_tch(4.6):C6014/81:delGen=22 _tk5(4.6):C3907/242:delGen=21
_tjv(4.6):C3492/119:delGen=13 _thd(4.6):C5014/241:delGen=24
_tdh(4.6):C5375/437:delGen=30 _tj1(4.6):C5989/15:delGen=6
_tkq(4.6):C1749/36:delGen=6 _tmj(4.6):C961/1:delGen=1
_tlm(4.6):C714/9:delGen=5 _tm6(4.6):C2616 _tlx(4.6):C1105/273:delGen=3
_tly(4.6):C5/2:delGen=1 _tm2(4.6):C1 _tm4(4.6):C1 _tmb(4.6):C1 _tmk(4.6):C5
_tml(4.6):C12 _tmm(4.6):C1 _tmn(4.6):C2/1:delGen=1 _tmo(4.6):C1 _tmp(4.6):C1
_tmr(4.6):C1 _tms(4.6):C1)}
2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO  org.apache.solr.core.SolrCore  – QuerySenderListener done.

2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO  org.apache.solr.handler.component.SpellCheckComponent  – Building
spell index for spell checker: suggest

2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO  org.apache.solr.spelling.suggest.Suggester  – build()/

Still no update...

/2014-08-12 13:12:58,424 | [81] | 154264021 [searcherExecutor-4-thread-1]
INFO  org.apache.solr.core.SolrCore  – [collection1] Registered new searcher
Searcher@65c48c06 main{StandardDirectoryReader(segments_2we:82217:nrt
_qkc(4.6):C8161558/879724:delGen=275 _sra(4.6):C2943436

RE: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

2014-07-25 Thread Matt Kuiper (Springblox) 
You might consider looking at your internal Solr cache configuration 
(solrconfig.xml).  These caches occupy heap space, and from my understanding do 
not overflow to disk.  So if there is not enough heap memory to support the 
caches an OOM error will be thrown.

I also believe these caches live in Old Gen.  So you might consider decreasing 
your CMSInitiatingOccupancyFraction to trigger a GC sooner.

Based on your description below every 20,000 documents your caches will be 
invalidated and rebuilt as part of a commit.  So a GC that occurs sooner may 
help free the memory of the old caches.  

Matt

-Original Message-
From: Ameya Aware [mailto:ameya.aw...@gmail.com] 
Sent: Friday, July 25, 2014 9:22 AM
To: solr-user@lucene.apache.org
Subject: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Hi,

I am in process of indexing lot of documents but after around 9 documents i 
am getting below error:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit

I am passing below parameters with Solr :

java -Xms6144m -Xmx6144m -XX:MaxPermSize=512m -Dcom.sun.management.jmxremote 
-XX:+UseParNewGC -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC 
-XX:+CMSIncrementalMode -XX:+CMSParallelRemarkEnabled 
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70 -XX:ConcGCThreads=6
-XX:ParallelGCThreads=6 -jar start.jar


Also, i am Auto-committing after 2 documents.


I searched on google for this but could not get any specific answer.


Can anybody help with this?


Thanks,
Ameya

RE: To warm the whole cache of Solr other than the only autowarmcount

2014-07-24 Thread Matt Kuiper (Springblox) 
I don't believe this would work.  My understanding (please correct if I have 
this wrong) is that the underlying Lucene document ids have a potential to 
change and so when a newSearcher is created the caches must be regenerated and 
not copied.

Matt

-Original Message-
From: YouPeng Yang [mailto:yypvsxf19870...@gmail.com] 
Sent: Thursday, July 24, 2014 10:26 AM
To: solr-user@lucene.apache.org
Subject: To warm the whole cache of Solr other than the only autowarmcount

Hi

   I think it is wonderful to have caches autowarmed when commit or soft commit 
happens. However ,If I want to warm the whole cache other than the only 
autowarmcount,the default the auto warming operation will take long long ~~long 
time.So it comes up with that maybe it good idea  to just change the reference 
of the caches of the newsearcher with the caches of the oldsearcher. It will 
increase the time of the autowarming,also increase the query time of NRT.
  It is just not a mature idea.I am pust this idea,and hope to get more hints 
or help to make more cleat about the idea.



regards

RE: Easises way to insatll solr cloud with tomcat

2014-05-13 Thread Matt Kuiper (Springblox) 
Check out http://heliosearch.com/download.html 

It is a distribution of Apache Solr packaged with Tomcat.

I have found it simple to use.

Matt

-Original Message-
From: Aman Tandon [mailto:amantandon...@gmail.com] 
Sent: Monday, May 12, 2014 6:24 AM
To: solr-user@lucene.apache.org
Subject: Re: Easises way to insatll solr cloud with tomcat

Can anybody help me out??

With Regards
Aman Tandon


On Mon, May 12, 2014 at 1:24 PM, Aman Tandon wrote:

> Hi,
>
> I tried to set up solr cloud with jetty which works fine. But in our 
> production environment we uses tomcat so i need to set up the solr 
> cloud with the tomcat. So please help me out to how to setup solr 
> cloud with tomcat on single machine.
>
> Thanks in advance.
>
> With Regards
> Aman Tandon
>