Is there way to autowarm new searcher using recently ran queries

Hi, A rookie question. We have a Solr cluster that doesn't get too much traffic. We see that our queries take long time unless we run a script to send more traffic to Solr. We are indexing data all the time and use autoCommit. I am wondering if there is a way to warmup new searcher on commit by

Add custom comparator for field(s) for Sorting.

Hi, Is it possible to add a custom comparator to a field for sorting. e.g. let's say I have field 'name' and following documents { id : "doc1", name : "1" } { id : "doc2", name : "S1" } { id : "doc2", name : "S2" } if I sort using field 'name', the order would be : ["doc1",

Is it possible to direct queries to replicas in SolrCloud

Hi, In master/slave we can send queries to slaves only, now that we have tlog and pull replicas can we send queries to those replicas to achieve similar scaling like master/slave for large search volumes? -- — Pushkar Raste

Re: Does Solr master/slave support shard split

d make a single-shard collection in SolrCloud, > copy the index to the right place (I’d shut down Solr while > I copied it), and then use SPLITSHARD on it, but that implies > you’d be going to SolrCloud. > > Best, > Erick > > > On May 21, 2020, at 10:35 AM, Pushkar Raste

Does Solr master/slave support shard split

Hi, Does Solr support shard split in the master/slave setup. I understand that there is no shard concept is master/slave and we just have cores but can we split a core into two. If yes is there way to specify new mapping based on the unique key. -- — Pushkar Raste

Re: Does Solr replicate data securely

Hi, Can some help me with my question. On Tue, Nov 12, 2019 at 10:20 AM Pushkar Raste wrote: > Hi, > How about in the master/slave set up. If I enable ssl in master/slave > setup would the segment and config files be copied using TLS. > > On Sat, Nov 9, 2019 at 3:31 PM Jan

Re: Does Solr replicate data securely

luding replication will be secure (https). This it is > still tcp but using TLS :) > > Jan Høydahl > > > 6. nov. 2019 kl. 00:03 skrev Pushkar Raste : > > > > Hi, > > When slaves/pull replicas copy index files from master is done using an > > secure protocol or just over tcp? > > -- > > — Pushkar Raste >

Does Solr replicate data securely

Hi, When slaves/pull replicas copy index files from master is done using an secure protocol or just over tcp? -- — Pushkar Raste

Re: What are the risk of running into "Unmap hack not supported on this platform"

at 3:08 PM Shawn Heisey wrote: > On 8/27/2019 8:22 AM, Pushkar Raste wrote: > > I am trying to run Solr 4 on JDK11, although this version is not > supported > > on JDK11 it seems to be working fine except for the error/exception > "Unmap > > hack not supported on th

Re: What are the risk of running into "Unmap hack not supported on this platform"

Can someone help me with this? On Tue, Aug 27, 2019 at 10:22 AM Pushkar Raste wrote: > Hi, > I am trying to run Solr 4 on JDK11, although this version is not supported > on JDK11 it seems to be working fine except for the error/exception "Unmap > hack not supported on this

What are the risk of running into "Unmap hack not supported on this platform"

Hi, I am trying to run Solr 4 on JDK11, although this version is not supported on JDK11 it seems to be working fine except for the error/exception "Unmap hack not supported on this platform". What the risks/downsides of running into this.

Re: Is it possible to reconstruct non stored fields and tun those into stored fields

but in this case > it’d be OK since they’re simple types) by examining the inverted index and > pulling out the values. Painfully slow and you’d have to write custom code > probably at the Lucene level to make it all work. > > Best, > Erick > > > On May 22, 2019, at 8:11 AM

Is it possible to reconstruct non stored fields and tun those into stored fields

if these fields can be retrieved while iterating over index while reading the documents. -- — Pushkar Raste

How to stop a new slave from serving request until it has replicated index the first time.

Hi, In the master/slave setup, as soon as I start a new slave it starts to serve request. Often the searches result in no documents being found as index has not been replicated yet. Is there a way to stop replica from serving request (marking node unhealthy) until the index is replicated for the

Re: API to convert a SolrInputDocument to JSON

: > https://issues.apache.org/jira/browse/SOLR-445 > > I have _not_ worked with this personally in prod SolrCloud systems, so > I can't say much more > than it exists. It's only available in Solr 6.1+ > > Best, > Erick > > On Wed, Jan 23, 2019 at 5:55 PM Pushkar Raste > w

Re: API to convert a SolrInputDocument to JSON

You mean I can use SolrJ 7.x for both indexing documents to both Solr 4 and Solr 7 as well as the SolrInputDocument class from Solrj 7.x Wouldn’t there be issues if there are any backwards incompatible changes. On Wed, Jan 23, 2019 at 8:09 PM Shawn Heisey wrote: > On 1/23/2019 5:49 PM, Push

Re: API to convert a SolrInputDocument to JSON

Thanks for the quick response Shawn. It is migrating ion from Solr 4.10 master/slave to Solr Cloud 7.x On Wed, Jan 23, 2019 at 7:41 PM Shawn Heisey wrote: > On 1/23/2019 5:05 PM, Pushkar Raste wrote: > > We are setting up cluster with new version Solr and going to reindex > data.

API to convert a SolrInputDocument to JSON

Hi, We are setting up cluster with new version Solr and going to reindex data. However, until all the data is indexed I need keep indexing data in the old cluster as well. We are currently using the Solrj client and constructing SolrInputDocument objects to index data. To avoid conflicts with

Re: Can Solr 4.10 work with JDK11

Or let me rephrase the question. What is the minimum Solr version that is JDK11 compatible. On Tue, Jan 15, 2019 at 10:27 AM Pushkar Raste wrote: > I probably already know the answer for this but was still wondering. >

Can Solr 4.10 work with JDK11

I probably already know the answer for this but was still wondering.

Questions about the IndexUpgrader tool.

Hi, I have questions about the IndexUpgrader tool. - I want to upgrade from Solr 4 to Solr 7. Can I run upgrade the index from 4 to 5 then 5 to 6 and finally 6 to 7 using appropriate version of the IndexUpgrader but without loading the Index in the Solr at all during the successive upgrades. -

Re: NullPointerException in PeerSync.handleUpdates

As mentioned in the JIRA, exception seems to be coming from a log statement. The issue was fixed in 6.3, here is relevant line f rom 6.3 https://github.com/apache/lucene-solr/blob/releases/lucene-solr/6.3.0/solr/core/src/java/org/apache/solr/update/PeerSync.java#L707 On Wed, Nov 22, 2017 at

Re: Spread SolrCloud across two locations

on't take any action when DC1 comes > back, we are still operational with 5 nodes quorum. Isn't it? Or I am > missing something. > > > > On Fri, May 26, 2017 at 10:07 AM, Pushkar Raste <pushkar.ra...@gmail.com> > wrote: > > > Damn, > > Math is hard

Re: Spread SolrCloud across two locations

Damn, Math is hard DC1 : 3 non observers DC2 : 2 non observers 3 + 2 = 5 non observers Observers don't participate in voting = non observers participate in voting 5 non observers = 5 votes In addition to the 2 non observer, DC2 also has an observer, which as you pointed out does not

Re: Spread SolrCloud across two locations

, search solution architect Cominvent AS - www.cominvent.com > 25. mai 2017 kl. 00.35 skrev Pushkar Raste <pushkar.ra...@gmail.com>: > > A setup I have used in the past was to have an observer I DC2. If DC1 one > goes boom you need manual intervention to change observer's role to

Re: Spread SolrCloud across two locations

A setup I have used in the past was to have an observer I DC2. If DC1 one goes boom you need manual intervention to change observer's role to make it a follower. When DC1 comes back up change on instance in DC2 to make it a observer again On May 24, 2017 6:15 PM, "Jan Høydahl"

Re: Disable All kind of caching in Solr/Lucene

What version are you on. There was a bug where if you use cache size 0, it would still create a cache with size 2 (or may be just 1). It was fixed under https://issues.apache.org/jira/browse/SOLR-9886?filter=-2 On Apr 3, 2017 9:26 AM, "Nilesh Kamani" wrote: > @Yonik

Re: Latest advice on G1 collector?

Hi Walter, We have been using G1GC for more than a year now and are very happy with it. The only flag we have enabled is 'ParallelRefProcEnabled' On Jan 23, 2017 3:00 PM, "Walter Underwood" wrote: > We have a workload with very long queries, and that can drive the CMS >

Re: retrieve ids of all indexed docs efficiently

I think we should add the suggestion about docValues to the cursormark wiki (documentation), we too ran in the same problem. On Jan 18, 2017 5:52 PM, "Erick Erickson" wrote: > Is your ID field docValues? Making it a docValues field should reduce > the amount of JVM heap

Re: SolrCloud: ClusterState says we are the leader but locally we don't think so

Try bouncing the overseer for your cluster. On Jan 17, 2017 12:01 PM, "Kelly, Frank" wrote: > Solr Version: 5.3.1 > > Configuration: 3 shards, 3 replicas each > > After running out of heap memory recently (cause unknown) we’ve been > successfully restarting nodes to

Re: Debug logging in Maven project

Seems like you have enabled only console appender. I remember there was a changed made to disable console appender if Solr is started in background mode. On Jan 10, 2017 5:55 AM, "Markus Jelsma" wrote: > Hello, > > I used to enable debug logging in my Maven project's

Re: Very long young generation stop the world GC pause

You should probably have as small a swap as possible. I still feel long GCs are either due to swapping or thread contention. Did you try to remove all other G1GC tuning parameters except for the ParallelRefProcEnabled? On Dec 19, 2016 1:39 AM, "forest_soup" wrote: >

Re: Separating Search and Indexing in SolrCloud

This kind of separation is not supported yet. There however some work going on, you can read about it on https://issues.apache.org/jira/browse/SOLR-9835 This unfortunately would not support soft commits and hence would not be a good solution for near real time indexing. On Dec 16, 2016 7:44

Re: Distribution Packages

We use jdeb maven plugin to build the debian packages, we use it for Solr as well On Dec 12, 2016 9:03 AM, "Adjamilton Junior" wrote: > Hi folks, > > I am new here and I wonder to know why there's no Solr 6.x packages for > ubuntu/debian? > > Thank you. > > Adjamilton Junior >

Re: Very long young generation stop the world GC pause

ntalVMOptions \ > -XX:G1MaxNewSizePercent=5 \ > > The aim here is to only limit the maximum and still allow some adaptation. > > --Ere > > 8.12.2016, 16.07, Pushkar Raste kirjoitti: > >> Disable all the G1GC tuning your are doing except for >> ParallelRefProcEnabled

Re: Very long young generation stop the world GC pause

Disable all the G1GC tuning your are doing except for ParallelRefProcEnabled G1GC is an adaptive algorithm and would keep tuning to reach the default pause goal of 250ms which should be good for most of the applications. Can you also tell us how much RAM you have on your machine and if you have

Re: AW: AW: Resync after restart

Did you index any documents while node was being restarted? There was a issue introduced due to IndexFingerprint comparison. Check SOLR-9310. I am not sure if fix made it to Solr6.2 On Nov 25, 2016 3:51 AM, "Arkadi Colson" wrote: > I am using SolrCloud on version 6.2.1. I

Re: Error "unexpected docvalues type NUMERIC for field" using rord() function query on single valued int field

Did you turn on/off docValues on a already existing field? On Nov 16, 2016 11:51 AM, "Jaco de Vroed" wrote: > Hi, > > I made a typo. The Solr version number in which this error occurs is 5.5.3. > I also checked 6.3.0, same problem. > > Thanks, bye, > > Jaco. > > On 16

Re: Merge policy

Try commit with expungeDeletes="true" I am not sure if it will merge old segments that have deleted documents. In the worst case you can 'optimize' your index which should take care of removing deleted document On Oct 27, 2016 4:20 AM, "Arkadi Colson" wrote: > Hi > > As

Re: Solr Cloud A/B Deployment Issue

Nodes will still go into recovery but only for a short duration. On Oct 26, 2016 1:26 PM, "jimtronic" wrote: It appears this has all been resolved by the following ticket: https://issues.apache.org/jira/browse/SOLR-9446 My scenario fails in 6.2.1, but works in 6.3 and

Re: Solr Cloud A/B Deployment Issue

This is due to leader initiated recovery. When Take a look at https://issues.apache.org/jira/browse/SOLR-9446 On Oct 24, 2016 1:23 PM, "jimtronic" wrote: > We are running into a timing issue when trying to do a scripted deployment > of > our Solr Cloud cluster. > >

Re: OOM Error

You should look into using docValues. docValues are stored off heap and hence you would be better off than just bumping up the heap. Don't enable docValues on existing fields unless you plan to reindex data from scratch. On Oct 25, 2016 3:04 PM, "Susheel Kumar" wrote: >

Re: OOM Error

Did you look into the heap dump ? On Mon, Oct 24, 2016 at 6:27 PM, Susheel Kumar wrote: > Hello, > > I am seeing OOM script killed solr (solr 6.0.0) on couple of our VM's > today. So far our solr cluster has been running fine but suddenly today > many of the VM's Solr

Re: disable updates during startup

The reason node is in recovery for long time could be related to https://issues.apache.org/jira/browse/SOLR-9310 On Tue, Oct 4, 2016 at 9:14 PM, Rallavagu wrote: > Solr Cloud 5.4.1 with embedded Jetty - jdk 8 > > Is there a way to disable incoming updates (from leader)

Re: group.facet fails when facet on double field

This error is thrown when you add (or remove) on an existing field but do not reindex you data from scratch. It is result of removing field cache from Lucene. Although you were not getting error with Solr 4.8, I am pretty sure that you were getting incorrect results. Stand up a small test

Re: Zookeeper connection issues

If Solr has GC pauses greater than 15 seconds, zookeeper is going to assume node is down and hence would send it into recovery when node comes out of a GC pause and reconnects to zookeeper. You should look into keeping GC pause as short as possible. Using G1GC with ParallelRefProcEnabled has

Re: solr perf metric for the last 1 hour (or last 10 mn)

Hi Dominique, Unfortunately Solr doesn't support metrics you are interested in. You can however have another process that makes jmx queries on the solr process, do required transformation and store data in some kind of data store. Just make sure you are not DDOSing your Solr instances :-) On Oct

Re: [Solr-5-4-1] Why SolrCloud leader is putting all replicas in recovery at the same time ?

A couple of questions/suggestions - This normally happens after leader election, when new leader gets elected, it will force all the nodes to sync with itself. Check logs to see when this happens, if leader was changed. If that is true then you will have to investigate why leader change takes

Re: Queries to help warm up (mmap)

One of the tricks I had read somewhere was to cat all files in the index directory and OS will have file in the disk cache. On Thu, Oct 6, 2016 at 11:55 AM, Rallavagu wrote: > Looking for clues/recommendations to help warm up during startup. Not > necessarily Solr caches

RE: how to sampling search result

Purely of algorithmic point of view - look into reservoir sampling for unbiased sampling. On Sep 28, 2016 11:00 AM, "Yongtao Liu" wrote: Alexandre, Thanks for reply. The use case is customer want to review document based on search result. But they do not want to review all,

Re: Whether SolrCloud can support 2 TB data?

Solr is RAM hungry. Make sure that you have enough RAM to have most if the index of a core in the RAM itself. You should also consider using really good SSDs. That would be a good start. Like others said, test and verify your setup. --Pushkar Raste On Sep 23, 2016 4:58 PM, "Jeffery

Re: Create collection PeerSync "no frame of reference" warnings

If you are creating a collection these warnings are harmless. There is patch being worked on under SOLR-9446 (although for a different scen) it would help suppressing this error.

RE: Facetting on a field doesn't work, until i optimized the index

Markus, Can you pick up one of the values in the facets and try running query using that. Ideally numFound should match the facet count. If those aren't matching I guess your index is still sorta damaged but you aren't really noticing it. On Sep 15, 2016 4:44 AM, "Markus Jelsma"

Re: Facetting on a field doesn't work, until i optimized the index

Damn I didn't put comments in the ticket but replied to question " Is it safe to upgrade an existing field to docvalues?" on the mailing list. Check that out On Sep 14, 2016 5:59 PM, "Pushkar Raste" <pushkar.ra...@gmail.com> wrote: > We experienced exact opposite is

Re: Facetting on a field doesn't work, until i optimized the index

We experienced exact opposite issue on Solr 4.10 Check my comments in https://issues.apache.org/jira/browse/SOLR-9437 I am not sure if issue was fixed in Solr 6 I do be interested in tracking down patch for this. On Sep 14, 2016 3:04 PM, "Erick Erickson" wrote: >

Re: Is it safe to upgrade an existing field to docvalues?

Hi Ronald, Turning on docValues for existing field works in Solr 4. As you mentioned it will use un-inverting method if docValues are nit found on existing document. This all works fine until segments that have documents without docValues merge with segment that have docValues for the field. In

Re: commit it taking 1300 ms

It would be worth looking into iostats of your disks. On Aug 22, 2016 10:11 AM, "Alessandro Benedetti" wrote: > I agree with the suggestions so far. > The cache auto-warming doesn't seem the problem as the index is not massive > and the auto-warm is for only 10 docs. >

Fwd: About SOLR-9310

-- Forwarded message -- From: "Pushkar Raste" <pushkar.ra...@gmail.com> Date: Jul 20, 2016 11:08 AM Subject: About SOLR-9310 To: <d...@lucene.apache.org> Cc: Hi, https://issues.apache.org/jira/browse/SOLR-9310 PeerSync replication in SOLR seems to be c

Re: SolrCloud: Frequent "No registered leader was found" errors

If you have GC logs, check if you have long GC pauses that make zookeeper think that node(s) are going down. If this is the cases then your nodes are going into recovery and and based on your settings in in solr.xml you may end up in situation when no nodes gets promoted to be a leader. On 22

Re: A field _indexed_at_tdt added when I index documents.

You must have this field in your schema with some default value assigned to it (most probably default value is NOW). This field is usually used to determine latest timestamp when this document was last indexed. On 17 December 2015 at 04:51, Guillermo Ortiz wrote: > I'm

Re: Solr 5.2.1 Most solr nodes in a cluster going down at once.

Hi Philippa, Try taking a heap dump (when heap usage is high) and then using a profiler look at which objects are taking up most of the memory. I have seen that if you are using faceting/sorting on large number of documents then fieldCache grows very big and dominates most of of the heap. Enabling

Re: Using properties placeholder ${someProperty} for xml node attribute in solrconfig

its size is > zero. > > Another option might be to add {!cache=false} to your fq clauses on > the client in this case if that is possible/convenient. > > Best, > Erick > > On Thu, Dec 3, 2015 at 11:19 AM, Pushkar Raste <pushkar.ra...@gmail.com> > wrote: > &g

Re: How to list all collections in solr-4.7.2

Will 'wget http://host;port//solr/admin/collections?action=LIST' help? On 3 December 2015 at 12:12, rashi gandhi wrote: > Hi all, > > I have setup two solr-4.7.2 server instances on two diff machines with 3 > zookeeper severs in solrcloud mode. > > Now, I want to

Using properties placeholder ${someProperty} for xml node attribute in solrconfig

Hi, I want to make turning filter cache on/off configurable (I really have a use case to turn off filter cache), can I use properties placeholders like ${someProperty} in the filter cache config. i.e. In short, can I use properties placeholders for attributes for xml node in solrconfig. Follow

Re: SolrCloud breaks and does not recover

HI, To minimize GC pauses, try using G1GC and turn on 'ParallelRefProcEnabled' jvm flag. G1GC works much better for heaps > 4 GB. Lowering 'InitiatingHeapOccupancyPercent' will also help to avoid long GC pauses at the cost of more short pauses. On 3 November 2015 at 12:12, Björn Häuser

Re: Is it impossible to update an index that is undergoing an optimize?

I may be wrong but I think 'delete' and 'optimize' can not be executed concurrently on a Lucene index On 4 November 2015 at 15:36, Shawn Heisey wrote: > On 11/4/2015 1:17 PM, Yonik Seeley wrote: > > On Wed, Nov 4, 2015 at 3:06 PM, Shawn Heisey > wrote:

How turn on logging for segment merging

. Is there special operation I have to set to turn on segment merging information? -- Pushkar Raste

Re: restore quorum after majority of zk nodes down

th 3 zk nodes on each side > > > > 2015-10-29 23:46 GMT+01:00 Pushkar Raste <pushkar.ra...@gmail.com>: > > > > > How about having let's say 4 nodes on each side and make one node in > one > > of > > > data centers a observer. When data cente

Re: restore quorum after majority of zk nodes down

outage as far as indexing is concerned but queries should continue to work and you don't have to take all the zookeeper nodes down. -- Pushkar Raste On Oct 29, 2015 4:33 PM, "Matteo Grolla" <matteo.gro...@gmail.com> wrote: > Hi Walter, > it's not a problem to take d

Re: Two seperate intance of Solr on the same machine

add "-Dsolr.log=" to your command line On 27 October 2015 at 08:13, Steven White <swhite4...@gmail.com> wrote: > How do I specify a different log directory by editing "log4j.properties"? > > Steve > > On Mon, Oct 26, 2015 at 9:08 PM, Pushka

Re: Two seperate intance of Solr on the same machine

It depends on your case. If you don't mind logs from 3 different instances inter-mingled with each other you should be fine. You add "-Dsolr.log=" to make logs to go different directories. If you want logs to go to same directory but different files try updating log4j.properties. On 26 October

Re: Anyone users IBM J9 JVM with 32G max heap ? Tuning recommendations?

Do you have GC logging turned on? If yes can you provide excerpt from the GC log for a pause that took > 30sec On 19 October 2015 at 04:16, Jeff Wu wrote: > Hi all, > > we are using solr4.7 on top of IBM JVM J9 Java7, max heap to 32G, system > RAM 64G. > > JVM parameters:

Re: slow queries

- turn on GC logging and see if there are any long and tune GC strategy accordingly. -- Pushkar Raste On Wed, Oct 14, 2015 at 5:03 AM, Lorenzo Fundaró < lorenzo.fund...@dawandamail.com> wrote: > Hello, > > I have following conf for filters and commits : > > Concurrent

Re: slow queries

Once you have GC logs, look for string "Total time for which application threads were stopped" to check if you have long pauses (you may get long pauses even with young generation GC). -- Pushkar Raste On Wed, Oct 14, 2015 at 11:47 AM, Lorenzo Fundaró < lorenzo.fund...@dawandama

Issue while adding Long.MAX_VALUE to a TrieLong field

Hi, I am trying to following add document (value for price.long is Long.MAX_VALUE) 411 one 9223372036854775807 However upon querying my collection value I get back for "price.long" is 9223372036854776000 Definition for 'price.long' field and 'long' look like

Re: Issue while adding Long.MAX_VALUE to a TrieLong field

see...@gmail.com> wrote: > On Thu, Sep 10, 2015 at 5:43 PM, Pushkar Raste <pushkar.ra...@gmail.com> > wrote: > > Did you see my previous response to you today? > http://markmail.org/message/wt6db4ocqmty5a42 > > Try querying a different way, like from the command line u

Issue while adding Long.MAX_VALUE to a TrieLong field

I am trying following add document (value for price.long is Long.MAX_VALUE) 411 one 9223372036854775807 However upon querying my collection value I get back for "price.long" is 9223372036854776000 (I got same behavior when I used JSON file) Definition for