Re: Are there roadblocks to creating custom DocRouter implementations?

Also interested in custom/pluggable routing. On Tue, May 16, 2017 at 4:47 PM, Erick Erickson wrote: > Hmmm, would the functionality be served by just using implicit routing > putting the logic in creating the doc and populating the route field? > Not, perhaps, as

Re: knowing which fields were successfully hit

Something like elasticsearch named-queries, right https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-named-queries-and-filters.html ? On Tue, May 16, 2017 at 7:10 PM, John Blythe wrote: > sorry for the confusion. as in i received results due to

Re: TrieIntField vs IntPointField performance only for equality comparison (no range filtering)

Hi Shawn, I forgot that legacy-int-fields were deprecated. Point fields it is then. Thanks, Dorian On Tue, May 16, 2017 at 3:01 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 5/16/2017 3:33 AM, Dorian Hoxha wrote: > > Has anyone measured which is more efficient/performan

TrieIntField vs IntPointField performance only for equality comparison (no range filtering)

Hi, Has anyone measured which is more efficient/performant between the 2 intfields if we don't need to do range-checking ? (precisionStep=0) Regards, Dorian

Re: Atomic Updates

mail.com > > > > wrote: > > > > > Hmm, interesting. I can imagine that as long as you're updating > > > docValues fields, the other_text field would be there. But the instant > > > you updated a non-docValues field (text_field in your example) the >

Re: Atomic Updates

t; > On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <dorian.ho...@gmail.com> > wrote: > > > You'll lose the data in that field. Try doing a commit and it should > > happen. > > > > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <culicny@iq.media> wrot

Re: Atomic Updates

You'll lose the data in that field. Try doing a commit and it should happen. On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny wrote: > Thanks Shawn, I didn't realize docValues were enabled by default now. > That's very convenient and probably makes a lot of the schemas we've been

Re: Return all docs with same last-value when sorting by non-unique-value

> Regards, >Alex > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 15 April 2017 at 08:06, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > Hi friends, > > > > Say we're sorting by a non-unique-value, and also

Return all docs with same last-value when sorting by non-unique-value

Hi friends, Say we're sorting by a non-unique-value, and also have a limit(x). But there are more docs in the end of list(x) that have the same value. Is it possible to return them even if the number of items will be > x ? This will make it possible so I don't have to sort by (non-unique,unique)

Re: Filtering results by minimum relevancy score

@alessandro Elastic-search has it: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-min-score.html On Wed, Apr 12, 2017 at 1:49 PM, alessandro.benedetti wrote: > I am not completely sure that the potential benefit of merging less docs in >

Re: Dynamic schema memory consumption

LR-7191 On Tue, Apr 11, 2017 at 9:11 PM, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > And this overhead depends on what? I mean, if I create an empty collection >> will it take up much heap size just for "being there" ? > > Yes. You can search on elastic-se

Re: Dynamic schema memory consumption

gt; have more than 50GB. > > > Dorian Hoxha wrote > > Each shard is a lucene index which has a lot of overhead. > > And this overhead depends on what? I mean, if I create an empty collection > will it take up much heap size just for "being there" ? > > > Dor

Re: Dynamic schema memory consumption

/dynamic memory-issue though. On Tue, Apr 11, 2017 at 6:09 PM, jpereira <jpereira...@gmail.com> wrote: > Dorian Hoxha wrote > > Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a > > little too much for 3TB of data ? > > Something like 0.167GB

Re: Filtering results by minimum relevancy score

Can't the filter be used in cases when you're paginating in sharded-scenario ? So if you do limit=10, offset=10, each shard will return 20 docs ? While if you do limit=10, _score<=last_page.min_score, then each shard will return 10 docs ? (they will still score all docs, but merging will be

Re: Dynamic schema memory consumption

Also you should change the heap 32GB->30GB so you're guaranteed to get pointer compression. I think you should have no need to increase it more than this, since most things have moved to out-of-heap stuff, like docValues etc. On Tue, Apr 11, 2017 at 12:07 PM, Dorian Hoxha <dorian.ho...@gma

Re: Dynamic schema memory consumption

Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a little too much for 3TB of data ? Something like 0.167GB for each shard ? Isn't that too much overhead (i've mostly worked with es but still lucene underneath) ? Can't you use 1/100 the current number of collections ? On

Re: SortingMergePolicy in solr 6.4.2

Did you get any update on this ? On Tue, Mar 14, 2017 at 11:56 AM, Sahil Agarwal wrote: > The SortingMergePolicy does not seem to get implemeted. > > The csv file gets indexed without errors. But when I search for a term, the > results returned are not sorted by

Str vs Int/Long uniqueKey field performance

Hey friends, Is there any difference on the index-size/performance of having the 'uniqueKey` field a long vs a string ? Meaning, does it use a different data-structure ? Cause I remember elasticsearch always uses a string (since it adds the #type, which solr doesn't have). Regards, Dorian

Re: Performance degradation after upgrading from 6.2.1 to 6.4.1

Did you see the other thread ? It looked like a problem with logging. On Tue, Feb 14, 2017 at 10:52 AM, Henrik Brautaset Aronsen < henrik.aron...@gmail.com> wrote: > We are seeing performance degradation on our SolrCloud instances after > upgrading to 6.4.1. > > > Here are a couple of graphs.

Re: solr-user-unsubscribe

Come on dude. Just look at instructions. Have a little respect. On Mon, Jan 30, 2017 at 1:55 PM, Rowe, William - 1180 - MITLL < william.r...@ll.mit.edu> wrote: > solr-user-unsubscribe > > >

Commit required after delete ?

Hello friends, Based on what I've read, I think "commit" isn't needed to make deletes active (like we do with index/update), right ? Since it just marks an in-memory deleted-id bitmap, right ? Thank You

Re: Cloud Behavior when using numShards=1

I think solr tries itself to load balance. Read this page https://cwiki.apache.org/confluence/display/solr/Distributed+Requests (preferLocalShards!) Also please write the query. tip: fill "send" address after completing email On Tue, Dec 27, 2016 at 4:31 PM, Dave Seltzer

Re: ttl on merge-time possible somehow ?

On Mon, Dec 19, 2016 at 7:03 PM, Chris Hostetter wrote: > > : So, the other way this can be made better in my opinion is (if the > : optimization is not already there) > : Is to make the 'delete-query' on ttl-documents operation on translog to > not > : be forced to

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

On Sun, Dec 18, 2016 at 3:48 PM, GW wrote: > Yeah, > > > I'll look at the proxy you suggested shortly. > > I've discovered that the idea of making a zookeeper aware app is pointless > when scripting REST calls right after I installed libzookeeper. > > Zookeeper is there to

Re: Soft commit and reading data just after the commit

There's a very high probability that you're using the wrong tool for the job if you need 1ms softCommit time. Especially when you always need it (ex there are apps where you need commit-after-insert very rarely). So explain what you're using it for ? On Sun, Dec 18, 2016 at 3:38 PM, Lasitha

Re: ttl on merge-time possible somehow ?

On Sat, Dec 17, 2016 at 12:04 AM, Chris Hostetter wrote: > > : > lucene, something has to "mark" the segements as deleted in order for > them > ... > : Note, it doesn't mark the "segment", it marks the "document". > > correct, typo on my part -- sorry. > > : >

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 10:53 PM, Chris Hostetter wrote: > > : Yep, that's what came in my search. See how TTL work in hbase/cassandra/ > : rocksdb . There > : isn't a "delete old docs"query, but old docs are

Re: ttl on merge-time possible somehow ?

. On Fri, Dec 16, 2016 at 9:57 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 12/16/2016 1:12 PM, Dorian Hoxha wrote: > > Shawn, I know how it works, I read the blog post. But I don't want it > > that > > way. So how to do it my way? Like a custom merge function on l

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 8:11 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 12/16/2016 11:13 AM, Dorian Hoxha wrote: > > Yep, that's what came in my search. See how TTL work in hbase/cassandra/ > > rocksdb <https://github.com/facebook/rocksdb/wiki/Time-to-Live>. Th

Re: Max vertical scaling in your experience ? (1 instance/server)

e a list of nodes ready for business verses a plain > > old Round Robin. If data on a machine is corrupted you can get 0 docs > found > > while a ZK aware app will know that node is shite. > > > > > > > > > > > > > > > > On 16 December 201

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 4:42 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 12/16/2016 12:54 AM, Dorian Hoxha wrote: > > I did some search for TTL on solr, and found only a way to do it with > > a delete-query. But that ~sucks, because you have to do a lot of >

Re: Separating Search and Indexing in SolrCloud

> > This is not a question of SolrClient cause that is clear how to direct > search request to specific nodes. This is more about index optimization > so that certain nodes (ie. replicas) could suffer less due to high > volume indexing while serving search requests. > > &g

Re: Separating Search and Indexing in SolrCloud

The leader is the source of truth. You expect to make the replica the source of truth or something???Doesn't make sense? What people do, is send write to leader/master and reads to replicas/slaves in other solr/other-dbs. On Fri, Dec 16, 2016 at 1:31 PM, Jaroslaw Rozanski

Re: Max vertical scaling in your experience ? (1 instance/server)

I commenter wrote on your thread. > > Will keep you posted on my roll out if you like. > > Best, > > GW > > > > > > > > On 16 December 2016 at 03:31, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > > Hello searchers, > > > > I'm re

Re: Max vertical scaling in your experience ? (1 instance/server)

On Fri, Dec 16, 2016 at 11:31 AM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > On Fri, 2016-12-16 at 11:19 +0100, Dorian Hoxha wrote: > > On Fri, Dec 16, 2016 at 10:45 AM, Toke Eskildsen > > <t...@statsbiblioteket.dk> wrote: > > > We try hard to

Re: Max vertical scaling in your experience ? (1 instance/server)

On Fri, Dec 16, 2016 at 10:45 AM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > On Fri, 2016-12-16 at 09:31 +0100, Dorian Hoxha wrote: > > I'm researching solr for a project that would require a max- > > inserts(10M/s) and some heavy facet+fq on top of that, though on

Max vertical scaling in your experience ? (1 instance/server)

Hello searchers, I'm researching solr for a project that would require a max-inserts(10M/s) and some heavy facet+fq on top of that, though on low qps. And I'm trying to find blogs/slides where people have used some big machines instead of hundreds of small ones. 1. Largest I've found is this

ttl on merge-time possible somehow ?

Hello searchers, I did some search for TTL on solr, and found only a way to do it with a delete-query. But that ~sucks, because you have to do a lot of inserts (and queries). The other(kinda better) way to do it, is to set a collection-level ttl, and when indexes are merged, they will drop the

Re: Search only for single value of Solr multivalue field

You should be able to filter "(word1 in field OR word2 in field) AND NOT(word1 in field AND word2 in field)". Translate that into the right syntax. I don't know if lucene is smart enough to execute the filter only once (it should be i guess). Makes sense ? On Thu, Dec 15, 2016 at 12:12 PM, Leo

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

y solr instances. > I've done that before (though with es, but it's ~same). And just contacting the localhost solr. The problem with that, is that if the solr on the current host fails, your php won't work. So best in this scenario is to have an array of hosts, but the first being the local sol

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

See replies inline: On Wed, Dec 14, 2016 at 11:16 AM, GW wrote: > Hello folks, > > I'm about to set up a Web service I created with PHP/Apache <--> Solr Cloud > > I'm hoping to index a bazillion documents. > ok , how many inserts/second ? > > I'm thinking about using

Re: Difference between currency fieldType and float fieldType

; > Thanks, > > Esther Quansah > > > Le 7 déc. 2016 à 07:26, Dorian Hoxha <dorian.ho...@gmail.com> a écrit : > > > > Come on dude, just use the int/long. > > Source: double is still a float. > > > > On Wed, Dec 7, 2016 at 1:17 PM, Zheng Lin Edwin Yeo <

Re: Difference between currency fieldType and float fieldType

o 32-bit for float. > But will it hit the same issue again if the amount exceeds 64-bit? > > Regards, > Edwin > > > On 7 December 2016 at 15:28, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > > Yeah, you'll have to do the conversion yourself (or something inte

Re: Difference between currency fieldType and float fieldType

Yeah, you'll have to do the conversion yourself (or something internal, like the currencyField). Think about it as datetimes. You store everything in utc (cents), but display to each user in it's own timezone (different currency, or just from cents to full dollars). On Wed, Dec 7, 2016 at 8:23

Re: Difference between currency fieldType and float fieldType

Don't use float for money (in whatever db). https://wiki.apache.org/solr/CurrencyField What you do is save the money as cents, and store that in a long. That's what the currencyField probably does for you inside. It provides currency conversion at query-time. On Tue, Dec 6, 2016 at 4:45 AM,

Re: Queries regarding solr cache

@Shawn Any idea why the cache doesn't use roaring bitsets ? On Thu, Dec 1, 2016 at 3:49 PM, Shawn Heisey wrote: > On 12/1/2016 4:04 AM, kshitij tyagi wrote: > > I am using Solr and serving huge number of requests in my application. > > > > I need to know how can I utilize

Realtime multi get with different (_route_, fields, etc) for each id

Hello searchers, Looks like this is not possible, right ? It means I have to specify all the _route_ in the request, and each shard will try to lookup all the ids internally. Is there a way to specify it ? Like elasticsearch does

Re: Index time sorting and per index mergePolicyFactory

bump after 11 days On Thu, Nov 17, 2016 at 10:25 AM, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > Hi, > > I know this is done in lucene, but I don't see it in solr (by searching + > docs on collections). > > I see https://cwiki.apache.org/confluence/display/solr/ >

Re: update a document without changing anything

Thanks Ishan. On Sun, Nov 27, 2016 at 6:42 AM, Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > Maybe do an "inc" of 0 to a numeric field for every document. > https://cwiki.apache.org/confluence/display/solr/ > Updating+Parts+of+Documents > > On Wed,

Re: Best python 3 client for solrcloud

s indexing, reindexing, etc.. > > Check it out and let me know what you think. > > On Nov 24, 2016 3:51 PM, "Dorian Hoxha" <dorian.ho...@gmail.com> wrote: > > > Hi searchers, > > > > I see multiple clients for solr in python but each one looks like mis

Best python 3 client for solrcloud

Hi searchers, I see multiple clients for solr in python but each one looks like misses many features. What I need is for at least the low-level api to work with cloud (like retries on different nodes and nice exceptions). What is the best that you use currently ? Thank You!

Re: Should zookeeper be run on the worker machines?

You can, but you should not. Source: heavy load may slow zookeeper resulting in timeouts etc. On Wed, Nov 23, 2016 at 5:00 PM, Tech Id wrote: > Hi, > > Can someone please respond to this zookeeper-for-Solr Stack-Overflow > question:

update a document without changing anything

Hello searcherers, So, I have document that is fully stored. Then I make small change in schema. And now I have to reinsert every document. But I'm afraid of doing a get+insert, because something else may change the document in the meantime. So I want to do an "update" of nothing, so internally

Re: Solr/lucene "planet" + recommendations for blogs to follow

ing list, but I'll figure it out. I > have enough info. > > Regards, >Alex. > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 22 November 2016 at 22:45, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > Thanks Alex,

Re: Solr/lucene "planet" + recommendations for blogs to follow

ctive: https://lucidworks.com/blog/ > > Regards, >Alex. > > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 22 November 2016 at 21:56, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > Hello searcherers, > > >

Re: Multiple search-queries in 1 http request ?

gt; >> > >> > >> [0] https://issues.apache.org/jira/browse/SOLR-1093 > >> > >> On Thu, Nov 17, 2016 at 7:50 PM, Mikhail Khludnev <m...@apache.org> > wrote: > >> > >>> Hello, > >>> There is nothing like that in Solr.

Solr/lucene "planet" + recommendations for blogs to follow

Hello searcherers, Is there a solr/lucene "planet" like planet.postgresql.org ? If not, what are some blogs/rss/feeds that I should follow to learn what's happening in the solr/lucene worlds ? Thank You

Re: Using solr(cloud) as source-of-truth for data (with no backing external db)

g if you are > wanting to save a DB lookup. > > I would still store everything in a DB though to have a "master" copy of > everthing. > > > > On 11/18/2016 04:45 AM, Dorian Hoxha wrote: > >> @alex >> That makes sense, but it can be ~fixed by just

Re: Bkd tree numbers/geo on solr 6.3 ?

Looks like it needs https://issues.apache.org/jira/browse/SOLR-8396 . On Thu, Nov 17, 2016 at 2:41 PM, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > Hi, > > I've read that lucene 6 has fancy bkd-tree implementation for numbers. But > on latest cwiki I only see T

Re: Using solr(cloud) as source-of-truth for data (with no backing external db)

is painful. > > > > Regards, > > Alex. > > > > Solr Example reading group is starting November 2016, join us at > > http://j.mp/SolrERG > > Newsletter and resources for Solr beginners and intermediates: > > http://www.solr-start.com/ > >

Changing route and still ending on the same shard (or increasing the % of shards a tenant is distrbuted on without reindexing)

Hi, Assuming I use `tenant1/4!doc50` for id (which means 1/16th of shards), and I later change it to `tenant1/2!doc50` (which means 1/8), is it guaranteed that the document will go to the same shard ? (it would be nice, but I don't think so). Meaning , when you change the `/x!`, do you have to

Bkd tree numbers/geo on solr 6.3 ?

Hi, I've read that lucene 6 has fancy bkd-tree implementation for numbers. But on latest cwiki I only see TrieNumbers. Aren't they implemented or did I miss something (they still mention "indexing multiple values for range-queries" , which is the old way)? Thank You

Multiple search-queries in 1 http request ?

Hi, I couldn't find anything in core for "multiple separate queries in 1 http request" like elasticsearch ? I found this blog-post

Updating documents with docvalues (not stored), commit question

Looks like you can update documents even using just doc-values (without stored). While I understand the columnar-format, my issue with this is that docValues are added when a 'commit' is done (right?). Does that mean that it will force a commit (which is a slow operation) when updating with

Re: Parent child relationship, where children aren't nested but separate (like elasticsearch)

gt; Solr Example reading group is starting November 2016, join us at > http://j.mp/SolrERG > Newsletter and resources for Solr beginners and intermediates: > http://www.solr-start.com/ > > > On 17 November 2016 at 18:08, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > &

Re: "add and limit" update modifier or scripted update like elasticsearch

roup is starting November 2016, join us at > http://j.mp/SolrERG > Newsletter and resources for Solr beginners and intermediates: > http://www.solr-start.com/ > > > On 17 November 2016 at 18:06, Dorian Hoxha <dorian.ho...@gmail.com> wrote: > > Hi, > > > > Is t

Index time sorting and per index mergePolicyFactory

Hi, I know this is done in lucene, but I don't see it in solr (by searching + docs on collections). I see https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig but it's not mentioned for index-time-sorting. So, is it possible and definable for each index ? I want to have

Using solr(cloud) as source-of-truth for data (with no backing external db)

Hi, Anyone use solr for source-of-data with no `normal` db (of course with normal backups/replication) ? Are there any drawbacks ? Thank You

How many versions do you stay behind in production for better stability ?

Hi, I see that there is a new release on every lucene release. Do you always use the latest version since it may have bugs (ex most cassandra productions are old compared to latest `stable` version because they're not stable). How much behind do you usually stay ? (ex: 6.3 just came out, and you

Parent child relationship, where children aren't nested but separate (like elasticsearch)

Hi, I'm not finding a way to support parent-child like es does (using blockjoin)? I've seen some blogs with having children as nested inside the parent-document, but I want to freely crud childs/parents as

"add and limit" update modifier or scripted update like elasticsearch

Hi, Is there an "add and limit" update modifier (couldn't find in docs) ? If not, can I run a script to update a document (still couldn't find anything) ? If not, how should I do that (custom plugin? )? Thank You

Re: book for Solr 3.4?

@HelponR Curious why you're interested in an old version ? On Tue, Nov 15, 2016 at 11:43 PM, HelponR wrote: > Thank you. Just found one here https://wiki.apache.org/solr/SolrResources > > "Apache Solr 3 Enterprise Search Server >