Re: Questions about Disk space Usage

2016-10-28 Thread Walter Underwood
After the merge. That is what merges do, clean up segments. I expect it is very rare for a segment to be 100% deleted docs, so it isn’t worth handling that case. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 28, 2016, at 5:54 PM, Alexandre Raf

Re: Questions about Disk space Usage

2016-10-28 Thread Alexandre Rafalovitch
Don't the segment that only has deleted documents just gets dropped? Or does it get dropped _after_ the merge and therefore still sits around? Regards, Alex. Solr Example reading group is starting November 2016, join us at http://j.mp/SolrERG Newsletter and resources for Solr beginners and

Re: Reindex after schema change options

2016-10-28 Thread Erick Erickson
This is a little contradictory: > how do I reindex the data in place - without starting from the source? > then ran my reindex SolrJ code. So it looks like you _were_ able to re-index from scratch? BTW, to be absolutely safe I'd re-index to a _new_ collection and then, perhaps, use collection a

Re: Questions about Disk space Usage

2016-10-28 Thread Walter Underwood
It is normal for disk usage to double. Under controlled circumstances, it can triple, but that probably won’t happen. This is the second time today that I’ve sent this information to the list. It can use nearly 2X the space whenever the largest segment(s) are merged, especially if there are only

Re: Questions about Disk space Usage

2016-10-28 Thread Alexandre Rafalovitch
2) Is probably a merge operation. Lucene index segments are not rewritable in place, so the merge creates a new file, does everything to it, then switches to it. I remember the number was that the space could temporarily triple (?!?) though that may have been before the tiered merge policy. 3) It

Questions about Disk space Usage

2016-10-28 Thread Jamal, Sarfaraz
Hi Guys, I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer. First Question * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte Is it safe to delete those file

Re: Merge policy

2016-10-28 Thread Walter Underwood
25% overhead is pretty good. It is easy for a merge to need almost double the space of a minimum sized index. It is possible to use 3X the space. Don’t try use the least possible disk space. If there isn’t enough free space on the disk, Solr cannot merge the big indexes. Ever. That may be what h

Re: Reindex after schema change options

2016-10-28 Thread tedsolr
So I ran a quick test of my idea and it worked. I modified the schema.xml file - uploaded it to ZK - reloaded the collection - then ran my reindex SolrJ code. After it completed the schema browser in the admin console shows that the field uses docValues. I tried a streaming expression on it to usin

Heatmap in JSON facet API

2016-10-28 Thread Никита Веневитин
Hi! Is it possible to use JSON facet API to get heatmaps?

Re: group.facet fails when facet on double field

2016-10-28 Thread karel braeckman
Thanks for your reply. I don't think this is what is causing the error though. We are starting from an empty index and have fully re-indexed a couple of documents. The query is still failing with the same error. On Sun, Oct 23, 2016 at 4:51 PM, Pushkar Raste wrote: > This error is thrown whe

sorting by date not working on dates earlier than EPOCH

2016-10-28 Thread marotosg
Hi all, I just noticed than sorting on dates is not working as I am expecting. When I sort ascending by a field of type date. I get first dates earlier than EPOCH, then null and last dates later than EPOCH. sample response: { numFound: 2052, start: 0, docs: [ { ClosedDateSFD: "1901-01-01T00:00:00

Re: Merge policy

2016-10-28 Thread Emir Arnautovic
I got some notification from mailer, so not sure if my reply reached you: "If you are using TieredMergePolicy, you can try setting /*reclaimDeletesWeight*/." HTH, Emir On 28.10.2016 09:20, Arkadi Colson wrote: The index size of 1 shard is about 125GB and we are running 11 shards with repl

Re: Comparing between 2 String fields

2016-10-28 Thread Emir Arnautovic
Hi Edwin, No - functions are introduced a long time ago - I think eq function was added in 6.x. Emir On 27.10.2016 18:05, Zheng Lin Edwin Yeo wrote: Hi Emir, Thanks for your reply. Does all these functions only works for Solr 6.x onwards? Regards, Edwin On 27 October 2016 at 20:13, Emir

Re: Merge policy

2016-10-28 Thread Arkadi Colson
It's a default installation using the default settings and parameters. Should I perhaps change the segment size or so? Is it possible to do live without re-indexing? If you need more info, just let me know... Thx! On 27-10-16 19:03, Walter Underwood wrote: That distribution of segment sizes

Re: Merge policy

2016-10-28 Thread Arkadi Colson
The index size of 1 shard is about 125GB and we are running 11 shards with replication factor 2 so it's a lot of data. The deletions percentage at the bottom of the segment page is around 25%. So it's quite some space which we could recover. That's why I was looking for an optimize. Do you ha