Re: dynamic field sorting

[Midas A]

waiting for reply . Actually Heap utilization increases when we sort with
dynamic fields

On Tue, Mar 21, 2017 at 10:37 AM, Midas A  wrote:

> Hi ,
>
> How can i improve the performance of dynamic field sorting .
>
> index size is : 20 GB
>
> Regards,
> Midas
>

Re: Solr Delete By Id Out of memory issue

[Erick Erickson]

Chris:

OK, The whole DBQ thing baffles the heck out of me so this may be
totally off base. But would committing help here? Or at least be worth
a test?

On Tue, Mar 21, 2017 at 4:28 PM, Chris Hostetter
 wrote:
>
> : Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
> : bounded by 1K count, but after looking at heap dump I was amazed how can it
> : keep more than 1K entries. But Yes I see around 7M entries according to
> : heap dump and around 17G of memory occupied by BytesRef there.
>
> what exactly are you looking at when you say you see "7M entries" ?
>
> are you sure you aren't confusing the keys in oldDeletes with other
> instances of BytesRef in the JVM?
>
> : It would be better to know why old deletes Map is used there. I am still
> : digging, If I find something then I will share that.
>
> See UpdateLog.lookupVersion for where the mapping is used if you are
> curious. Replicas recieving updates need to be able to verify (and in some
> casees ask the leader via RTG) for the version used when a doc was deleted
> in the even of concurrent/out-of-order updates for the same uniqueKey.
>
>
>
> -Hoss
> http://www.lucidworks.com/

Re: model building

[Tim Casey]

Joe,

To do this correctly, soundly, you will need to sample the data and mark
them as threatening or neutral.  You can probably expand on this quite a
bit, but that would be a good start.  You can then draw another set of
samples and see how you did.  You use one to train and one to validate.

What you are doing is probably just noise, from a model point of view, and
it will probably not make too much difference how you index/query/model
through the noise.

I don't mean this critically, just plainly.  Effectively the less
mathematically correctly you do this process, the more anecdotal the result.

tim


On Mon, Mar 20, 2017 at 4:42 PM, Joel Bernstein  wrote:

> I've only tested with the training data in it's own collection, but it was
> designed for multiple training sets in the same collection.
>
> I suspect you're training set is too small to get a reliable model from.
> The training sets we tested with were considerably larger.
>
> All the idfs_ds values being the same seems odd though. The idfs_ds in
> particular were designed to be accurate when there are multiple training
> sets in the same collection.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Mon, Mar 20, 2017 at 5:41 PM, Joe Obernberger <
> joseph.obernber...@gmail.com> wrote:
>
> > If I put the training data into its own collection and use q="*:*", then
> > it works correctly.  Is that a requirement?
> > Thank you.
> >
> > -Joe
> >
> >
> >
> > On 3/20/2017 3:47 PM, Joe Obernberger wrote:
> >
> >> I'm trying to build a model using tweets.  I've manually tagged 30
> tweets
> >> as threatening, and 50 random tweets as non-threatening.  When I build
> the
> >> mode with:
> >>
> >> update(models2, batchSize="50",
> >>  train(UNCLASS,
> >>   features(UNCLASS,
> >>  q="ProfileID:PROFCLUST1",
> >>  featureSet="threatFeatures3",
> >>  field="ClusterText",
> >>  outcome="out_i",
> >>  positiveLabel=1,
> >>  numTerms=250),
> >>   q="ProfileID:PROFCLUST1",
> >>   name="threatModel3",
> >>   field="ClusterText",
> >>   outcome="out_i",
> >>   maxIterations="100"))
> >>
> >> It appears to work, but all the idfs_ds values are identical. The
> >> terms_ss values look reasonable, but nearly all the weights_ds are 1.0.
> >> For out_i it is either -1 for non-threatening tweets, and +1 for
> >> threatening tweets.  I'm trying to follow along with Joel Bernstein's
> >> excellent post here:
> >> http://joelsolr.blogspot.com/2017/01/deploying-ai-alerting-s
> >> ystem-with-solrs.html
> >>
> >> Tips?
> >>
> >> Thank you!
> >>
> >> -Joe
> >>
> >>
> >
>

Re: Error with polygon search

[David Smiley]

Hello Hank,

The online version of the reference guide is always for the latest Solr 
release.  I think your configuration would work in the latest release.  Prior 
to Solr 6, the Spatial4J library had a different Java package location: replace 
"org.locationtech.spatial4j" with "com.spatial4j.core".  The only JTS jar file 
you need is "jts-1.14.jar".

~ David

> On Mar 21, 2017, at 4:31 PM, hank  wrote:
> 
> Hello,
> 
> 
> I'm having problems with a polygon search on location data. I've tried to 
> enable the JTS and Polygons from 
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search but I get the 
> following error when I load solr
> 
> 
>   java.util.concurrent.ExecutionException: 
> org.apache.solr.common.SolrException: Unable to create core [jordan]
>  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>  at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:496)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>  at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.solr.common.SolrException: Unable to create core 
> [jordan]
>  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
>  at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:87)
>  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:467)
>  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:458)
>  ... 5 more
> Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
> stats: Can't load schema 
> /opt/solr/solr-5.5.2/server/solr/stats/conf/managed-schema: Plugin 
> Initializing failure for [schema.xml] fieldType
>  at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:84)
>  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812)
>  ... 8 more
> Caused by: org.apache.solr.common.SolrException: Can't load schema 
> /opt/solr/solr-5.5.2/server/solr/stats/conf/managed-schema: Plugin 
> Initializing failure for [schema.xml] fieldType
>  at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:577)
>  at org.apache.solr.schema.IndexSchema.(IndexSchema.java:159)
>  at 
> org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
>  at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173)
>  at 
> org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:47)
>  at 
> org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:70)
>  at 
> org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:108)
>  at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:79)
>  ... 9 more
> Caused by: org.apache.solr.common.SolrException: Plugin Initializing failure 
> for [schema.xml] fieldType
>  at 
> org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:194)
>  at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:470)
>  ... 16 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory
>  at 
> com.spatial4j.core.context.SpatialContextFactory.makeSpatialContext(SpatialContextFactory.java:100)
>  at 
> org.apache.solr.schema.AbstractSpatialFieldType.init(AbstractSpatialFieldType.java:119)
>  at 
> org.apache.solr.schema.AbstractSpatialPrefixTreeFieldType.init(AbstractSpatialPrefixTreeFieldType.java:55)
>  at 
> org.apache.solr.schema.SpatialRecursivePrefixTreeFieldType.init(SpatialRecursivePrefixTreeFieldType.java:37)
>  at org.apache.solr.schema.FieldType.setArgs(FieldType.java:174)
>  at 
> org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:150)
>  at 
> org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:53)
>  at 
> org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:191)
>  ... 17 more
> Caused by: java.lang.ClassNotFoundException: 
> org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  at 
> com.spatial4j.core.context.SpatialContextFactory.makeSpatialContext(SpatialContextFactory.java:97)
>  ... 24 more
> 
> 
> 
> My field looks like
> 
> 
> class="solr.SpatialRecursivePrefixTreeFieldType"
>   
> spatialContextFactory="org.loca

Exception while integrating openNLP with Solr

[aruninfo100]

Hi,

I am trying to integrate openNLP with Solr.

The fieldtype is :

  
  


   
  


en-lemmatizer.txt->The file has a size close to 5mb.
I am using the lemmatizer dictionary from below link:

https://raw.githubusercontent.com/richardwilly98/elasticsearch-opennlp-auto-tagging/master/src/main/resources/models/en-lemmatizer.dict

  
field schema:



When I try to index I get the following error:

error :Error from server at http://localhost:8983/solr/star: Exception
writing document id 578df0de-6adc-4ca2-9d5d-23c5c088f83a to the index;
possible analysis error.

solr.log:


2017-03-22 00:03:42.477 INFO  (qtp1389647288-14) [   x:star]
o.a.s.u.p.LogUpdateProcessorFactory [star]  webapp=/solr path=/update
params={wt=javabin&version=2}{} 0 116
2017-03-22 00:03:42.478 ERROR (qtp1389647288-14) [   x:star]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception
writing document id 303e190b-b02c-4927-9669-733e76164f61 to the index;
possible analysis error.
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:939)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1094)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:720)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
at
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
at 
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:274)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
at 
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:239)
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:157)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
at 
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
at
org.ap

Re: Solr Delete By Id Out of memory issue

[Chris Hostetter]

: Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
: bounded by 1K count, but after looking at heap dump I was amazed how can it
: keep more than 1K entries. But Yes I see around 7M entries according to
: heap dump and around 17G of memory occupied by BytesRef there.

what exactly are you looking at when you say you see "7M entries" ?

are you sure you aren't confusing the keys in oldDeletes with other 
instances of BytesRef in the JVM?

: It would be better to know why old deletes Map is used there. I am still
: digging, If I find something then I will share that.

See UpdateLog.lookupVersion for where the mapping is used if you are 
curious. Replicas recieving updates need to be able to verify (and in some 
casees ask the leader via RTG) for the version used when a doc was deleted 
in the even of concurrent/out-of-order updates for the same uniqueKey.



-Hoss
http://www.lucidworks.com/

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Hi Chris,

Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
bounded by 1K count, but after looking at heap dump I was amazed how can it
keep more than 1K entries. But Yes I see around 7M entries according to
heap dump and around 17G of memory occupied by BytesRef there.

It would be better to know why old deletes Map is used there. I am still
digging, If I find something then I will share that.

Thanks
Rohit


On Tue, Mar 21, 2017 at 4:00 PM, Chris Hostetter 
wrote:

>
> : facing. We are storing messages in solr as documents. We are running a
> : pruning job every night to delete old message documents. We are deleting
> : old documents by calling multiple delete by id query to solr. Document
> : count can be in millions which we are deleting using SolrJ client. We are
> : using delete by id because it is faster than delete by query. It works
> : great for few days but after a week these delete by id get accumulated in
> : Linked hash map of UpdateLog (variable name as olddeletes). Once this map
> : is full then we are seeing out of memory.
>
> first off: what version of Solr are you running?
>
> UpdateLog.oldDeletes is bounded at numDeletesToKeep=1000 entries -- any
> more then that and the oldest entry is automatically deleted when more
> items are added.  So it doesn't really make sense to me that you would be
> seeing OOMs from this map filling up endlessly.
>
> Are you seeing more then 1000 entries in this map when you look at your
> heap dumps?
>
> : I am not sure why it is keeping the reference of all old deletes.
>
> It's complicated -- the short answer is that it's protection against out
> of order updates ariving from other nodes in SolrCloud under highly
> concurrent updates.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Solr Delete By Id Out of memory issue

[Chris Hostetter]

: facing. We are storing messages in solr as documents. We are running a
: pruning job every night to delete old message documents. We are deleting
: old documents by calling multiple delete by id query to solr. Document
: count can be in millions which we are deleting using SolrJ client. We are
: using delete by id because it is faster than delete by query. It works
: great for few days but after a week these delete by id get accumulated in
: Linked hash map of UpdateLog (variable name as olddeletes). Once this map
: is full then we are seeing out of memory.

first off: what version of Solr are you running?

UpdateLog.oldDeletes is bounded at numDeletesToKeep=1000 entries -- any 
more then that and the oldest entry is automatically deleted when more 
items are added.  So it doesn't really make sense to me that you would be 
seeing OOMs from this map filling up endlessly.

Are you seeing more then 1000 entries in this map when you look at your 
heap dumps?

: I am not sure why it is keeping the reference of all old deletes.

It's complicated -- the short answer is that it's protection against out 
of order updates ariving from other nodes in SolrCloud under highly 
concurrent updates.



-Hoss
http://www.lucidworks.com/

Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Hi All,

I am looking for some help to solve an out of memory issue which we are
facing. We are storing messages in solr as documents. We are running a
pruning job every night to delete old message documents. We are deleting
old documents by calling multiple delete by id query to solr. Document
count can be in millions which we are deleting using SolrJ client. We are
using delete by id because it is faster than delete by query. It works
great for few days but after a week these delete by id get accumulated in
Linked hash map of UpdateLog (variable name as olddeletes). Once this map
is full then we are seeing out of memory.

We did look into heap dump then we found that this map is storing BytesRef
as key and LogPtr as value. BytesRef is the one which is taking a lot of
memory. BytesRef is storing reference of all the ids which we are deleting.
I am not sure why it is keeping the reference of all old deletes.

I looked at solr code but I could not trace how it is cleaning this map.
There is a deleteAll method in UpdateLog and only test cases are calling
this method.

Did anyone face the same issue? I really appreciate a reply for this
problem.

*Note*: We are running Solr in cloud and because of this there is high gc
pause which is causing first replica go in recovery then leader and replica
crashes.

-
Rohit

Error with polygon search

[hank]

Hello,


I'm having problems with a polygon search on location data. I've tried to 
enable the JTS and Polygons from 
https://cwiki.apache.org/confluence/display/solr/Spatial+Search but I get the 
following error when I load solr


   java.util.concurrent.ExecutionException: 
org.apache.solr.common.SolrException: Unable to create core [jordan]
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:496)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [jordan]
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
  at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:87)
  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:467)
  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:458)
  ... 5 more
Caused by: org.apache.solr.common.SolrException: Could not load conf for core 
stats: Can't load schema 
/opt/solr/solr-5.5.2/server/solr/stats/conf/managed-schema: Plugin Initializing 
failure for [schema.xml] fieldType
  at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:84)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812)
  ... 8 more
Caused by: org.apache.solr.common.SolrException: Can't load schema 
/opt/solr/solr-5.5.2/server/solr/stats/conf/managed-schema: Plugin Initializing 
failure for [schema.xml] fieldType
  at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:577)
  at org.apache.solr.schema.IndexSchema.(IndexSchema.java:159)
  at 
org.apache.solr.schema.ManagedIndexSchema.(ManagedIndexSchema.java:104)
  at 
org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173)
  at 
org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:47)
  at 
org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:70)
  at 
org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:108)
  at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:79)
  ... 9 more
Caused by: org.apache.solr.common.SolrException: Plugin Initializing failure 
for [schema.xml] fieldType
  at 
org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:194)
  at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:470)
  ... 16 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory
  at 
com.spatial4j.core.context.SpatialContextFactory.makeSpatialContext(SpatialContextFactory.java:100)
  at 
org.apache.solr.schema.AbstractSpatialFieldType.init(AbstractSpatialFieldType.java:119)
  at 
org.apache.solr.schema.AbstractSpatialPrefixTreeFieldType.init(AbstractSpatialPrefixTreeFieldType.java:55)
  at 
org.apache.solr.schema.SpatialRecursivePrefixTreeFieldType.init(SpatialRecursivePrefixTreeFieldType.java:37)
  at org.apache.solr.schema.FieldType.setArgs(FieldType.java:174)
  at 
org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:150)
  at 
org.apache.solr.schema.FieldTypePluginLoader.init(FieldTypePluginLoader.java:53)
  at 
org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:191)
  ... 17 more
Caused by: java.lang.ClassNotFoundException: 
org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at 
com.spatial4j.core.context.SpatialContextFactory.makeSpatialContext(SpatialContextFactory.java:97)
  ... 24 more



My field looks like





And I've downloaded the jar files from 
https://repo1.maven.org/maven2/com/vividsolutions/jts-core/1.14.0/ and placed 
them in /opt/solr/solr-5.5.2/server/solr-webapp/webapp/WEB-INF/lib


jts-1.14.jar*
jts-1.14-sources.jar*
jts-core-1.14.0.jar*
jtsio-1.14.jar*
jtsio-1.14-sources.jar*

I'm running solr version 5.5.2


I've restarted solr several times and I keep getting an error that it doesn't 
know where org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory is, 
do I need to do something else with the jar files?


Thanks

Re: Error creating core [da]: Error opening new searcher

[Erick Erickson]

Well, I always chmod 777 when developing on my laptop ;)

And I agree it's much easier to get it all working with a sledgehammer
_then_ play with getting permissions correct. At least then I know
what changed last.



On Tue, Mar 21, 2017 at 11:51 AM, David Hastings
 wrote:
> This is true, I was speaking more from a development standpoint as if im
> the only one with access to the machine.
>
> On Tue, Mar 21, 2017 at 2:47 PM, Erick Erickson 
> wrote:
>
>> "Should you be aware of anything". Yes. Your security guys will go
>> nuts if you recursively gave 777 permissions. By changing this you've
>> opened up your directories to anyone with any access to add any
>> program to any directory and execute it. From a security standpoint
>> this is very bad practice.
>>
>> To fix the permissions in a more controlled fashion, you should
>> _install_ Solr as the same user who _runs_ solr. That'll straighten
>> out your permissions early in the process. Alternatively, add the user
>> who installs Solr _and_ the user who Solr runs as in the same Unix
>> group and permit the directories to that group.
>>
>> Best
>> Erick
>>
>> On Tue, Mar 21, 2017 at 11:29 AM, HrDahl  wrote:
>> > When I recursively gave permissions to all folders/files under
>> /opt/bitnami
>> > it worked! So now I can actually see my core aswell as use it.
>> >
>> > What when I go into production? Should I be aware of anything?
>> >
>> >
>> >
>> > --
>> > View this message in context: http://lucene.472066.n3.
>> nabble.com/Error-creating-core-da-Error-opening-new-
>> searcher-tp4326041p4326085.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>>

Re: Error creating core [da]: Error opening new searcher

[David Hastings]

This is true, I was speaking more from a development standpoint as if im
the only one with access to the machine.

On Tue, Mar 21, 2017 at 2:47 PM, Erick Erickson 
wrote:

> "Should you be aware of anything". Yes. Your security guys will go
> nuts if you recursively gave 777 permissions. By changing this you've
> opened up your directories to anyone with any access to add any
> program to any directory and execute it. From a security standpoint
> this is very bad practice.
>
> To fix the permissions in a more controlled fashion, you should
> _install_ Solr as the same user who _runs_ solr. That'll straighten
> out your permissions early in the process. Alternatively, add the user
> who installs Solr _and_ the user who Solr runs as in the same Unix
> group and permit the directories to that group.
>
> Best
> Erick
>
> On Tue, Mar 21, 2017 at 11:29 AM, HrDahl  wrote:
> > When I recursively gave permissions to all folders/files under
> /opt/bitnami
> > it worked! So now I can actually see my core aswell as use it.
> >
> > What when I go into production? Should I be aware of anything?
> >
> >
> >
> > --
> > View this message in context: http://lucene.472066.n3.
> nabble.com/Error-creating-core-da-Error-opening-new-
> searcher-tp4326041p4326085.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Error creating core [da]: Error opening new searcher

[Erick Erickson]

"Should you be aware of anything". Yes. Your security guys will go
nuts if you recursively gave 777 permissions. By changing this you've
opened up your directories to anyone with any access to add any
program to any directory and execute it. From a security standpoint
this is very bad practice.

To fix the permissions in a more controlled fashion, you should
_install_ Solr as the same user who _runs_ solr. That'll straighten
out your permissions early in the process. Alternatively, add the user
who installs Solr _and_ the user who Solr runs as in the same Unix
group and permit the directories to that group.

Best
Erick

On Tue, Mar 21, 2017 at 11:29 AM, HrDahl  wrote:
> When I recursively gave permissions to all folders/files under /opt/bitnami
> it worked! So now I can actually see my core aswell as use it.
>
> What when I go into production? Should I be aware of anything?
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Error-creating-core-da-Error-opening-new-searcher-tp4326041p4326085.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Grouping and result pagination

[Shawn Heisey]

On 3/21/2017 10:34 AM, Shawn Heisey wrote:
> Restating the original problem:  I cannot paginate through the groups
> in a grouped query.  The first page works, subsequent pages do not.  I
> have a distributed index.  Co-locating documents in the same group
> onto the same shard is going to require a complete redesign of
> indexing.  It's something that could be done, but not without a LOT of
> work.

Strange thing ... now when I try a paginated query, it works.  I have no
idea what I was doing differently before when it wasn't working.

solr-impl version:
4.9-SNAPSHOT 1680667 - solr - 2015-05-20 14:23:11

I have discovered that I can't get the query to work at all on 6.3.0
with my schema even without pagination.  I've encountered this bug again:

https://issues.apache.org/jira/browse/SOLR-8088

Thanks,
Shawn

Re: Error creating core [da]: Error opening new searcher

[HrDahl]

When I recursively gave permissions to all folders/files under /opt/bitnami
it worked! So now I can actually see my core aswell as use it. 

What when I go into production? Should I be aware of anything?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Error-creating-core-da-Error-opening-new-searcher-tp4326041p4326085.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Error creating core [da]: Error opening new searcher

[David Hastings]

Here:
/opt/bitnami/apache-solr/server/solr/da/data/index/write.lock

so just set the permissions on /opt/bitnami/ to 777 it will save you
headaches later.  you can use the ap but not make any indexes.  also delete
that file if it exists

On Tue, Mar 21, 2017 at 1:16 PM, HrDahl  wrote:

> The question is what permissions ? Is it read/write permissions on the
> write.lock file?
>
> And the solr is running and I can access the admin panel via the url port
> given by google cloud, I just cant add a core.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Error-creating-core-da-Error-opening-new-
> searcher-tp4326041p4326065.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Error creating core [da]: Error opening new searcher

[HrDahl]

The question is what permissions ? Is it read/write permissions on the
write.lock file?

And the solr is running and I can access the admin panel via the url port
given by google cloud, I just cant add a core.




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Error-creating-core-da-Error-opening-new-searcher-tp4326041p4326065.html
Sent from the Solr - User mailing list archive at Nabble.com.

dataimport to a smaller Solr farm

[deansg]

Hello,
My team often uses the /dataimport & /dih handlers to move items from one
Solr collection to another. However, all the times we did that, the number
of shards in the new collection was always the same or higher than in the
old. 
Can /dataimport work if I have less shards in the new collection than in the
old one? I tried specifying a shard from the new collection multiple times
in the data-config file, and it didn't seem to work - there were no visible
exceptions, but most items simply didn't enter the new collection.
Dean.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-to-a-smaller-Solr-farm-tp4326067.html
Sent from the Solr - User mailing list archive at Nabble.com.

ObjectId datatype handling in SOLR 5.4.0

[Abhijit Pawar]

Hello,

I am trying to index fields with ObjectId datatype from my mongoDB into
SOLR 5.4.0 using DIH(DataImportHandler) however it does not allow me to
index it.
I have to change it to string in my mongodb and then index it.

Is there a way to index ObjectId datatype fields in SOLR schema ?


Best Regards,

​Abhijit​

Re: Grouping and result pagination

[Shawn Heisey]

On 3/17/2017 9:26 AM, Shawn Heisey wrote:

On 3/17/2017 9:07 AM, Erick Erickson wrote:
"group.ngroups and group.facet require that all documents in each 
group must be co-located on the same shard in order for accurate 
counts to be returned."
That is not how things work right now. The index has 170 million 
documents in it, split into six large cold shards and a very small hot 
shard.


Restating the original problem:  I cannot paginate through the groups in 
a grouped query.  The first page works, subsequent pages do not.  I have 
a distributed index.  Co-locating documents in the same group onto the 
same shard is going to require a complete redesign of indexing.  It's 
something that could be done, but not without a LOT of work.


Should it be considered a bug that this doesn't work at all?  I call it 
a bug.  I'd be OK with being told that performance of paginated queries 
with grouping is terrible on a distributed index, but I'd like it to at 
least function.


Thanks,
Shawn

Re: Error creating core [da]: Error opening new searcher

[Erick Erickson]

Permissions would be my first guess.

Second guess: you killed Solr somehow (crash, kill -9 etc) and it left
the write.lock file laying around. Shut down Solr and manually remove
it.

Best,
Erick

On Tue, Mar 21, 2017 at 9:12 AM, David Hastings
 wrote:
> AccessDeniedException
>
>
>
> im going to guess permissions, open them up when youre developing, well, i
> do, to 777
>
> On Tue, Mar 21, 2017 at 11:37 AM, HrDahl  wrote:
>
>> I am trying to create a solr core on a google cloud linux server using
>> binami
>> launchpad. But when im trying to create a new core it gives me the below
>> error message.
>>
>> Can anyone see what is wrong?
>>
>> org.apache.solr.common.SolrException: Error opening new searcher
>> at org.apache.solr.core.SolrCore.(SolrCore.java:952)
>> at org.apache.solr.core.SolrCore.(SolrCore.java:816)
>> at org.apache.solr.core.CoreContainer.create(
>> CoreContainer.java:890)
>> at org.apache.solr.core.CoreContainer.create(
>> CoreContainer.java:827)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.lambda$
>> static$0(CoreAdminOperation.java:88)
>> at
>> org.apache.solr.handler.admin.CoreAdminOperation.execute(
>> CoreAdminOperation.java:377)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.
>> call(CoreAdminHandler.java:379)
>> at
>> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(
>> CoreAdminHandler.java:165)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> RequestHandlerBase.java:166)
>> at
>> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
>> HttpSolrCall.java:664)
>> at org.apache.solr.servlet.HttpSolrCall.call(
>> HttpSolrCall.java:445)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:345)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:296)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1691)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:143)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(
>> SecurityHandler.java:548)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.
>> doHandle(SessionHandler.java:226)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1180)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.
>> doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1112)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:213)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.
>> handle(HandlerCollection.java:119)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:320)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:251)
>> at
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
>> AbstractConnection.java:273)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:95)
>> at
>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
>> SelectChannelEndPoint.java:93)
>> at
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> executeProduceConsume(ExecuteProduceConsume.java:303)
>> at
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> produceConsume(ExecuteProduceConsume.java:148)
>> at
>> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
>> ExecuteProduceConsume.java:136)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:671)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:589)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.solr.common.SolrException: Error opening new
>> searcher
>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:
>> 1891)
>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
>> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
>> at org.apache.solr.core.SolrCore.(SolrCore.java:925)
>> ... 37 more
>> Caused by: java.nio.file.AccessDeniedException:
>> /opt/bitnami/apache-solr/server/solr/da/data/index/write.lock
>> at sun

Re: Error creating core [da]: Error opening new searcher

[David Hastings]

AccessDeniedException



im going to guess permissions, open them up when youre developing, well, i
do, to 777

On Tue, Mar 21, 2017 at 11:37 AM, HrDahl  wrote:

> I am trying to create a solr core on a google cloud linux server using
> binami
> launchpad. But when im trying to create a new core it gives me the below
> error message.
>
> Can anyone see what is wrong?
>
> org.apache.solr.common.SolrException: Error opening new searcher
> at org.apache.solr.core.SolrCore.(SolrCore.java:952)
> at org.apache.solr.core.SolrCore.(SolrCore.java:816)
> at org.apache.solr.core.CoreContainer.create(
> CoreContainer.java:890)
> at org.apache.solr.core.CoreContainer.create(
> CoreContainer.java:827)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.lambda$
> static$0(CoreAdminOperation.java:88)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(
> CoreAdminOperation.java:377)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.
> call(CoreAdminHandler.java:379)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(
> CoreAdminHandler.java:165)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:166)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> HttpSolrCall.java:664)
> at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:445)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:345)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:296)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.solr.common.SolrException: Error opening new
> searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:
> 1891)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
> at org.apache.solr.core.SolrCore.(SolrCore.java:925)
> ... 37 more
> Caused by: java.nio.file.AccessDeniedException:
> /opt/bitnami/apache-solr/server/solr/da/data/index/write.lock
> at sun.nio.fs.UnixException.translateToIOException(
> UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(
> UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(
> UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(
> UnixFileSystemProvider.java:177)
> at java.nio.channels.FileChanne

Error creating core [da]: Error opening new searcher

[HrDahl]

I am trying to create a solr core on a google cloud linux server using binami
launchpad. But when im trying to create a new core it gives me the below
error message.

Can anyone see what is wrong?

org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.(SolrCore.java:952)
at org.apache.solr.core.SolrCore.(SolrCore.java:816)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:379)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:165)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
at org.apache.solr.core.SolrCore.(SolrCore.java:925)
... 37 more
Caused by: java.nio.file.AccessDeniedException:
/opt/bitnami/apache-solr/server/solr/da/data/index/write.lock
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at
org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:113)
at 
org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at 
org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at
org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104)

Re: Seek Help

[Erick Erickson]

You haven't really told us _how_ you are indexing, so I'm going to
make some comments that may be irrelevant...

At 600M documents, you'll almost certainly have to shard your index.
It sounds like you're doing the sharding yourself in Lucene by having
different Lucene indexes based on date. As you indicate, this makes it
more difficult for users. SolrCloud handles this for you and is very
likely preferable.

bq:  I know that SolrCloud can solve the search problem when the index
data is big, but it`s even slower in indexing than Solr.

This is not necessarily true. In fact, SolrCloud should be much faster
_if_ you use SolrJ, and the "CloudSolrClient" in the SolrJ program to
index to Solr. Under the covers, Solr routes documents to the correct
shard based on a hash of the . CloudSolrClient sends the
documents to the correct shard automatically, so if you have 10 shards
and index a batch of, say, 1,000 documents, 10 groups of 100 docs will
be sent out in parallel, one to each shard.

Here's an example of a SolrJ program (just a Java program using Solr
libraries): https://lucidworks.com/2012/02/14/indexing-with-solrj/.
Note that this code is rather old so it uses StreamingUpdateSolrServer
where you should use CloudSolrClient. It also processes structured
documents using Tika, but you can remove those bits of the code.

One technique I use when using SolrJ is to comment out the single line
that sends the doc to Solr (_server.add(docs) in the example). This
tells me whether the bottleneck is in getting the data from the
database or indexing it to Solr. Often the bottleneck is getting the
data, but with 600M documents that may not be the case.

Once your cluster is set up, you might then be able to fire up several
indexing clients. This assumes that you can partition getting the data
from your database. Say you are indexing 10 years' of data. Fire up 10
clients each of which only indexes 1 year's worth of data.

Hope that helps,
Erick


On Tue, Mar 21, 2017 at 7:08 AM, Q&Q <793555...@qq.com> wrote:
> Dear  Sir/Madam, I am Li Wei, from China, and I`m writing to you for your 
> help. Here is the problem I encountered:
>
>
> There is a timing task set at night in our project which uses  Lucene to 
> build index for the data from Oracle database. It was working fine at the 
> beginning, however, as the index file grows bigger, the indexing work is 
> getting slower, when the data needed to be indexed is big, the timing task 
> can't be finished at night.
>
>
> To solve this problem, we take the following measure:
> We store the index data in different directories according to the time the 
> data inserted into the database. This measure can solve the indexing problem 
> in some way. However, when searching the index data, the user has to specify 
> the year when the data is created so as to search in the corresponding 
> directory，it`s a bad experience for the users.
>
>
> Then we learned that Solr is good at indexing data from database, so we 
> decide to adopt Solr into our project. But as the index data gets bigger, it 
> would also take more and more time for Solr to finish the index task. I know 
> that SolrCloud can solve the search problem when the index data is big, but 
> it`s even slower in indexing than Solr.
>
>
> So I am writing to you for help. Is there any solution for Solr to handle 
> this kind problem? There are more than six hundred million records in the 
> database right now, and data will be added into the database everyday. 
> Whether it is true that if we don't set the UniqueKey property in the 
> config.xml file, then the problem will be avoided? If so, there`s another 
> problem, the index data can be only added, but can't be updated without the 
> UniqueKey property. Could you please give me some solutions for these 
> problems?
>
>
> I am looking forward to you sincerely. Thank you very much for your time!
>
>
> Best regards,
> Li Wei

Seek Help

Dear  Sir/Madam, I am Li Wei, from China, and I`m writing to you for your help. 
Here is the problem I encountered:


There is a timing task set at night in our project which uses  Lucene to build 
index for the data from Oracle database. It was working fine at the beginning, 
however, as the index file grows bigger, the indexing work is getting slower, 
when the data needed to be indexed is big, the timing task can't be finished at 
night. 


To solve this problem, we take the following measure:
We store the index data in different directories according to the time the data 
inserted into the database. This measure can solve the indexing problem in some 
way. However, when searching the index data, the user has to specify the year 
when the data is created so as to search in the corresponding directory??it`s a 
bad experience for the users.


Then we learned that Solr is good at indexing data from database, so we decide 
to adopt Solr into our project. But as the index data gets bigger, it would 
also take more and more time for Solr to finish the index task. I know that 
SolrCloud can solve the search problem when the index data is big, but it`s 
even slower in indexing than Solr.


So I am writing to you for help. Is there any solution for Solr to handle this 
kind problem? There are more than six hundred million records in the database 
right now, and data will be added into the database everyday. Whether it is 
true that if we don't set the UniqueKey property in the config.xml file, then 
the problem will be avoided? If so, there`s another problem, the index data can 
be only added, but can't be updated without the UniqueKey property. Could you 
please give me some solutions for these problems?


I am looking forward to you sincerely. Thank you very much for your time!


Best regards,
Li Wei

Re: block join - search together at parent and childern

[Mikhail Khludnev]

Hello Jan,
If I get you right, you need to find every word either in parent or child
level, hence:

q=+({!edismax qf=$pflds v=$w1} {!parent ..}{!edismax qf=$cflds v=$w1})
+({!edismax qf=$pflds v=$w1} {!parent ..}{!edismax qf=$cflds
v=$w1})...&w1=foo&w2=bar
note that spaces and + matter much. This yields cross-matches, but you
probably don't bother about them.

On Sun, Mar 19, 2017 at 11:58 AM, Jan Nekuda  wrote:

> Hi Michael,
> thank you for fast answer - I have tried it, but it's not exactly what I
> need. I hope that I understood it good - the problem is that if I will
> write foo bar and foo bar is not found in root entity then it returns
> nothing even if any field in children contains foo bar.
> I need to write foo bar and find all documents where foo bar exists in
> document A OR B OR C OR D even if in A will have FOO and in e.g C will be
> bar. But if I will write bar of chocolate then I need return nothing.
>
> my idea was to use
> edismax and filter query for each word:
> http://localhost:8983/solr/demo/select?q=*:*&fq={!parent
> which=type:root}foo*&fq={!parent
> which=typ:root}bar*&wt=json&indent=true&defType=edismax&
> qf=$allfields&stopwords=true&lowercaseOperators=true&allfieldscolor,
> first_country, power, name, country
>
> the problem is that I'm not able to find also parent documents in one
> condition with children.
>
> How I wrote I'm able solve it with another parent and then also doc A will
> be child and everything will work fine - but I would like to solve it
> better.
>
>
> Do you have or someone else another idea?:)
>
> Thanks
> Jan
>
>
> 2017-03-16 21:51 GMT+01:00 Mikhail Khludnev :
>
> > Hello Jan,
> >
> > What if you combine child and parent dismaxes like below
> > q={!edismax qf=$parentfields}foo bar {!parent ..}{!dismax qf=$childfields
> > v=$childclauses}&childclauses=foo bar +type:child&parentfields=...&
> > parentfields=...
> >
> > On Thu, Mar 16, 2017 at 10:54 PM, Jan Nekuda 
> wrote:
> >
> > > Hello Mikhail,
> > >
> > > thanks for fast answer. The problem is, that I want to have the dismax
> on
> > > child and parent together - to have the filter evaluated together.
> > >
> > > I need to have documents:
> > >
> > >
> > > path: car
> > >
> > > type:car
> > >
> > > color:red
> > >
> > > first_country: CZ
> > >
> > > name:seat
> > >
> > >
> > >
> > > path: car\engine
> > >
> > > type:engine
> > >
> > > power:63KW
> > >
> > >
> > >
> > > path: car\engine\manufacturer
> > >
> > > type:manufacturer
> > >
> > > name: xx
> > >
> > > country:PL
> > >
> > >
> > > path: car
> > >
> > > type:car
> > >
> > > color:green
> > >
> > > first_country: CZ
> > >
> > > name:skoda
> > >
> > >
> > >
> > > path: car\engine
> > >
> > > type:engine
> > >
> > > power:88KW
> > >
> > >
> > >
> > > path: car\engine\manufacturer
> > >
> > > type:manufacturer
> > >
> > > name: yy
> > >
> > > country:PL
> > >
> > >
> > > where car is parent document engine is its child a manufacturer is
> child
> > > of engine and the structure can be deep.
> > >
> > > I need to make a query with edismax over fields color, first_country,
> > > power, name, country over parent and all childern.
> > >
> > > when I ask then "seat 63 kw" i need to get seat car
> > >
> > > the same if I will write only "seat" or only "63kw" or only "xx"
> > >
> > > but if I will write "seat 88kw" i expect that i will get no result
> > >
> > > I need to return parents in which tree are all the words which I wrote
> to
> > > query.
> > >
> > > How I wrote before my solution was to split the query text and use
> q:*:*
> > > and for each /word/ in query make
> > >
> > > fq={!parent which=type:car}/word//
> > > /
> > >
> > > //and edismax with qf=color, first_country, power, name, country
> > >
> > > Thank you for your time:)
> > >
> > > Jan
> > >
> > >
> > > Dne 16.03.2017 v 20:00 Mikhail Khludnev napsal(a):
> > >
> > >
> > > Hello,
> > >>
> > >> It's hard to get into the problem. but you probably want to have
> dismax
> > on
> > >> child level:
> > >> q={!parent ...}{!edismax qf='childF1 childF2' v=$chq}&chq=foo bar
> > >> It's usually broken because child query might match parents which is
> not
> > >> allowed. Thus, it's probably can solved by adding +type:child into
> chq.
> > >> IIRC edismax supports lucene syntax.
> > >>
> > >> On Thu, Mar 16, 2017 at 4:47 PM, Jan Nekuda 
> > wrote:
> > >>
> > >> Hi,
> > >>> I have a question for which I wasn't able to find a good solution.
> > >>> I have this structure of documents
> > >>>
> > >>> A
> > >>> |\
> > >>> | \
> > >>> B \
> > >>>   \
> > >>>C
> > >>> \
> > >>>  \
> > >>>   \
> > >>>D
> > >>>
> > >>> Document type A has fields id_number, date_from, date_to
> > >>> Document type C  has fields first_name, surname, birthdate
> > >>> Document type D AND B has fields street_name, house_number, city
> > >>>
> > >>>
> > >>> I want to find *all parents with block join and edismax*.
> > >>> The problem is that I have found that possible is find children b