Re: Frequent deletions

2015-01-12 Thread ig01
Hi, We gave 120G to JVM, while we have 140G memory on this machine. We use the default merge policy(TieredMergePolicy), and there are 54 segments in our index. We tried to perform an optimization with different numbers of maxSegments (53 and less) it didn't help. How much memory we need for 180G

Distributed unit tests and SSL doesn't have a valid keystore

2015-01-12 Thread Markus Jelsma
Hi - in a small Maven project depending on Solr 4.10.3, running unit tests that extend BaseDistributedSearchTestCase randomly fail with SSL doesn't have a valid keystore, and a lot of zombie threads. We have a solrtest.keystore file laying around, but where to put it? Thanks, Markus

Re: solrcloud nodes registering as 127.0.1.1

2015-01-12 Thread Michael Della Bitta
Another way of doing it is by setting the -Dhost=$hostname parameter when you start Solr. Michael Della Bitta Senior Software Engineer o: +1 646 532 3062 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY 10017 t: @appinions https://twitter.com/Appinions |

solrcloud nodes registering as 127.0.1.1

2015-01-12 Thread Matteo Grolla
Hi, hope someone can help me troubleshoot this issue. I'm trying to setup a solrcloud cluster with -zookeeper on 192.168.1.8 (osx mac) -solr1 on 192.168.1.10 (virtualized ubuntu running on mac) -solr2 on 192.168.1.3 (ubuntu on another pc) the problem is

Re: solrcloud nodes registering as 127.0.1.1

2015-01-12 Thread Matteo Grolla
Solved! ubuntu has an entry like this in /etc/hosts 127.0.1.1 hostname to properly run solrcloud one must substitute 127.0.1.1 with a real (possibly permanent) ip address Il giorno 12/gen/2015, alle ore 12:47, Matteo Grolla ha scritto: Hi, hope someone can help me

Re: leader split-brain at least once a day - need help

2015-01-12 Thread Thomas Lamy
Hi, I found no big/unusual GC pauses in the Log (at least manually; I found no free solution to analyze them that worked out of the box on a headless debian wheezy box). Eventually i tried with -Xmx8G (was 64G before) on one of the nodes, after checking allocation after 1 hour run time was

How to apply SOLR-6024 to Solr 4.8

2015-01-12 Thread Elran Dvir
Hi all, I am trying to apply SOLR-6024 patch to Solr 4.8. I have some compilation errors with it (detailed in Jira: https://issues.apache.org/jira/i#browse/SOLR-6024). How can I change the patch to be applied to 4.8? Thanks.

ApacheCon 2015 at Austin, TX

2015-01-12 Thread CP Mishra
Hi, I am planning to attend ApacheCon 2015 at Austin, TX (Apr 13-16th) and wondering if there will be lucene/solr sessions in it. Anyone else planning to attend? Thanks, CP

Re: Why suggestions can be that slow?

2015-01-12 Thread FiMko
Seems I know the answer. The example query from mentioned above page: http://localhost:8983/solr/techproducts/suggest?suggest=truesuggest.build=truesuggest.dictionary=mySuggesterwt=jsonsuggest.q=elec

Why suggestions can be that slow?

2015-01-12 Thread FiMko
Hi all, I'm experimenting with Solr Suggester https://cwiki.apache.org/confluence/display/solr/Suggester . I have configured the functionality as per the mentioned page. In my Solr collection I have 32607 documents. The SuggestComponent is configured to search suggestions through field of type

Re: Why suggestions can be that slow?

2015-01-12 Thread Erick Erickson
Don't build it on every invocation. You only need to build the suggester when a new searcher is opened, i.e. omit suggest.build=true Best, Erick On Mon, Jan 12, 2015 at 7:31 AM, FiMko fima-regi...@ya.ru wrote: Hi all, I'm experimenting with Solr Suggester

Re: Problem with getting node active

2015-01-12 Thread O. Klein
UpdateLog got commented. Problem solved. -- View this message in context: http://lucene.472066.n3.nabble.com/Problem-with-getting-node-active-tp4178942p4179013.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Custom plugin classloader issue

2015-01-12 Thread Chris Hostetter
:I am stuck at a strange issue, I have my custom Query Component that has : to load spring application context for some additional runtime filtering of : records. :I have included my jars as dependency in solrConfig.xml, SOLR is able to : load my plugin but spring appplication fails to

Re: Frequent deletions

2015-01-12 Thread Shawn Heisey
On 1/10/2015 11:46 PM, ig01 wrote: Thank you all for your response, The thing is that we have 180G index while half of it are deleted documents. We tried to run an optimization in order to shrink index size but it crashes on ‘out of memory’ when the process reaches 120G. Is it possible to

Re: filter on solr pivot data

2015-01-12 Thread Darniz
Thanks for the reply but a filter query like -[* TO *] will give me vins which dont have a photo, it might qualify a dealer to show up but what if that dealer has other vin which has photo my requirement is i want to show the dealer only if all vin have no photos -- View this message in

RE: Extending solr analysis in index time

2015-01-12 Thread Markus Jelsma
Hi - You mention having a list with important terms, then using payloads would be the most straightforward i suppose. You still need a custom similarity and custom query parser. Payloads work for us very well. M -Original message- From:Ahmet Arslan iori...@yahoo.com.INVALID Sent:

Custom plugin classloader issue

2015-01-12 Thread Mohmed Hussain
Hi All, I am stuck at a strange issue, I have my custom Query Component that has to load spring application context for some additional runtime filtering of records. I have included my jars as dependency in solrConfig.xml, SOLR is able to load my plugin but spring appplication fails to load

Re: Unexplained leader initiated recovery after updates

2015-01-12 Thread Lindsay Martin
Here are more details about our setup: Zookeeper: * 3 separate hosts in same rack as Solr cluster * Zookeeper hosts do not run any other processes Solr: * total servers: 24 (plus 2 cold standbys in case of host failure) * physical memory: 65931872 kB (62 GB) * max JVM heap size: -Xmx10880m ( 10

Re: How to configure Solr PostingsFormat block size

2015-01-12 Thread Michael Sokolov
It looks like this is a good starting point: http://wiki.apache.org/solr/SolrConfigXml#codecFactory -Mike On 01/12/2015 03:37 PM, Tom Burton-West wrote: Hello all, Our indexes have around 3 billion unique terms, so for Solr 3, we set TermIndexInterval to about 8 times the default. The net

Re: Custom plugin classloader issue

2015-01-12 Thread Mohmed Hussain
Thanks Chris, that worked. Loaded my Spring Application context with plugin class loader. Was trying to resolve this since a day, and you resolved it in a minute :) Thanks -Hussain On Mon, Jan 12, 2015 at 11:29 AM, Chris Hostetter hossman_luc...@fucit.org wrote: :I am stuck at a strange

How to configure Solr PostingsFormat block size

2015-01-12 Thread Tom Burton-West
Hello all, Our indexes have around 3 billion unique terms, so for Solr 3, we set TermIndexInterval to about 8 times the default. The net effect of this is to reduce the size of the in-memory index by about 1/8th. (For background see for

Re: How to configure Solr PostingsFormat block size

2015-01-12 Thread Chris Hostetter
: It looks like this is a good starting point: : : http://wiki.apache.org/solr/SolrConfigXml#codecFactory The default SchemaCodecFactory already supports defining a diff posting format per fieldType - but there isn't much in solr to let you tweak individual options on specific posting formats

get one document per value in multivalued field

2015-01-12 Thread vit
I use Solr4.21 my multivalued filed is like that: q=(category_id:(484986 520623 484339 519258 516227 486757) .. How to construct a query which will show one top document per category_id value? -- View this message in context:

Re: get one document per value in multivalued field

2015-01-12 Thread Shawn Heisey
On 1/12/2015 3:09 PM, vit wrote: I use Solr4.21 my multivalued filed is like that: q=(category_id:(484986 520623 484339 519258 516227 486757) .. How to construct a query which will show one top document per category_id value? This is a feature called grouping, or field collapsing.

Re: get one document per value in multivalued field

2015-01-12 Thread vit
The field must be single-valued for grouping. That is why I do not consider this option. -- View this message in context: http://lucene.472066.n3.nabble.com/get-one-document-per-value-in-multivalued-field-tp4179056p4179065.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Unexplained leader initiated recovery after updates

2015-01-12 Thread Lindsay Martin
I have uncovered some additional details in the shard leader log: 2015-01-11 09:38:00.693 [qtp268575911-3617101] INFO org.apache.solr.update.processor.LogUpdateProcessor – [listings] webapp=/solr path=/update params{distrib.from=http://solr05.search.abebooks.com:8983/solr/listings/u

Understanding SolrCloud Restart Behavior - 4.6 onwards

2015-01-12 Thread KNitin
Hi I am trying to understand the process/node restart flow in a SolrCloud Cluster . What are the exact set of steps occur (like core/collection recovery, zk interaction etc) when a node is getting restarted? I am looking to implement some business logic at a collection/node level when solr is

Beachhead

2015-01-12 Thread William Bell
Using Amazon Ec2, we are using one machine to replicate to other instances in the Region. Instead of using 8GB of RAM, is there a way to replicate and use a LOT less memory? Would like to use t2.medium... Thoughts? -- Bill Bell billnb...@gmail.com cell 720-256-8076

Re: Beachhead

2015-01-12 Thread Shawn Heisey
On 1/12/2015 11:52 PM, William Bell wrote: Using Amazon Ec2, we are using one machine to replicate to other instances in the Region. Instead of using 8GB of RAM, is there a way to replicate and use a LOT less memory? Would like to use t2.medium... Can you provide more details about your

Solr limiting number of rows to indexed to 21500 every time.

2015-01-12 Thread Pankaj Sonawane
Hi, I am using Solr DataImportHandler to index data from database table(Oracle). One of the column contains String representation of XML (Sample below). *options* *option name=A1/option* *option name=B2/option* *option name=C3/option* *.* *.* *.* */options //option can be 100-200* I want

Re: Frequent deletions

2015-01-12 Thread ig01
Hi, Unfortunately this is the case, we do have hundreds of millions of documents on one Solr instance/server. All our configs and schema are with default configurations. Our index size is 180G, does that mean that we need at least 180G heap size? Thanks. -- View this message in context:

Re: SolrCloud shard leader elections - Altering zookeeper sequence numbers

2015-01-12 Thread Erick Erickson
Just skimming, but the problem here that I ran into was with the listeners. Each _Solr_ instance out there is listening to one of the ephemeral nodes (the one in front). So deleting a node does _not_ change which ephemeral node the associated Solr instance is listening to. So, for instance, when

Re: Distributed unit tests and SSL doesn't have a valid keystore

2015-01-12 Thread Mark Miller
I'd have to do some digging. Hossman might know offhand. You might just want to use @SupressSSL on the tests :) - Mark On Mon Jan 12 2015 at 8:45:11 AM Markus Jelsma markus.jel...@openindex.io wrote: Hi - in a small Maven project depending on Solr 4.10.3, running unit tests that extend

Problem with getting node active

2015-01-12 Thread O. Klein
I have 4 cores, of which 2 recover just fine and 2 others never get really active. Not when deleting index or changing clusterstate.json. So I created a new collection (1 shard, 2 replicas on Solr 4.5 with 3 zookeeper ensemble) and added 1 document to it. It never gets active. Not even on leader.

Re: leader split-brain at least once a day - need help

2015-01-12 Thread Mark Miller
bq. ClusterState says we are the leader, but locally we don't think so Generally this is due to some bug. One bug that can lead to it was recently fixed in 4.10.3 I think. What version are you on? - Mark On Mon Jan 12 2015 at 7:35:47 AM Thomas Lamy t.l...@cytainment.de wrote: Hi, I found no

SolrCloud shard leader elections - Altering zookeeper sequence numbers

2015-01-12 Thread Zisis Tachtsidis
SolrCloud uses ZooKeeper sequence flags to keep track of the order in which nodes register themselves as leader candidates. The node with the lowest sequence number wins as leader of the shard. What I'm trying to do is to keep the leader re-assignments to the minimum during a rolling restart. In

RE: Determining the Number of Solr Shards

2015-01-12 Thread Andrew Butkus
We decided to downgrade to 20 shards again, as we kept having the query time spikes, if it was a memory issue, I would assume we would have the same performance issues with 20 shards, so I think this is maybe a problem in solr rather than our configuration / amount of ram. In anycase, we have

Re: Extending solr analysis in index time

2015-01-12 Thread Jack Krupansky
Could you clarify what you mean by Lucene reverse index? That's not a term I am familiar with. -- Jack Krupansky On Mon, Jan 12, 2015 at 1:01 AM, Ali Nazemian alinazem...@gmail.com wrote: Dear Jack, Thank you very much. Yeah I was thinking of function query for sorting, but I have to

Re: How to apply SOLR-6024 to Solr 4.8

2015-01-12 Thread Shawn Heisey
On 1/12/2015 4:20 AM, Elran Dvir wrote: I am trying to apply SOLR-6024 patch to Solr 4.8. I have some compilation errors with it (detailed in Jira: https://issues.apache.org/jira/i#browse/SOLR-6024). How can I change the patch to be applied to 4.8? The compile errors seem to indicate that

Re: Why suggestions can be that slow?

2015-01-12 Thread FiMko
Erick, thanks for the answer. You're absolutely right! -- View this message in context: http://lucene.472066.n3.nabble.com/Why-suggestions-can-be-that-slow-tp4178944p4179007.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Extending solr analysis in index time

2015-01-12 Thread Ahmet Arslan
Hi Ali, Reading your example, if you could somehow replace idf component with your importance weight, I think your use case looks like TFIDFSimilarity. Tf component remains same. https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html I also suggest