***UNCHECKED*** Re: Re: solr 7.0: What causes the segment to flush

[DIMA]

Buongiorno,




Vedi allegato e di confermare.

Password: 1234567




Grazie




DIMA





From: khi...@gmail.com
Sent: Tue, 17 Oct 2017 15:40:50 +
To: solr-user@lucene.apache.org
Subject: Re: solr 7.0: What causes the segment to flush
 

I take my yesterdays comment back. I assumed that the file being written
is a segment, however after letting solr run for the night. I see that the
segment is flushed at the expected size:1945MB (so that file which i
observed was still open for writing).
Now, I have two other questions:-

1. Is there a way to not write to disk continuously and only write the file
when segment is flushed?

2. With 6.5: i had ramBufferSizeMB=20G and limiting the threadCount to 12
(since LUCENE-6659
,
there is no configuration for indexing thread count, so I did a local
workaround to limit the number of threads in code); I had very good write
throughput. But with 7.0, I am getting comparable throughput only at
indexing threadcount > 50. What could be wrong ?


Thanks @Erick, I checked the commit settings, both soft and hard commits
are off.




On Tue, Oct 17, 2017 at 3:47 AM, Amrit Sarkar 
wrote:

> >
> > In 7.0, i am finding that the file is written to disk very early on
> > and it is being updated every second or so. Had something changed in 7.0
> > which is causing it?  I tried something similar with solr 6.5 and i was
> > able to get almost a GB size files on disk.
>
>
> Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting
> 20GB segments on 6.5 or less? a GB?
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>
> On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal 
> wrote:
>
> > Hi,
> >
> > I have  tuned  (or tried to tune) my settings to only flush the segment
> > when it has reached its maximum size. At the moment,I am using my
> > application with only a couple of threads (i have limited to one thread
> for
> > analyzing this scenario) and my ramBufferSizeMB=2 (i.e. ~20GB). With
> > this, I assumed that my file sizes on the disk will be at in the order of
> > GB; and no segments will be flushed until the segments in memory size 
>is
> > 2GB. In 7.0, i am finding that the file is written to disk very early on
> > and it is being updated every second or so. Had something changed in 7.0
> > which is causing it?  I tried something similar with solr 6.5 and i was
> > able to get almost a GB size files on disk.
> >
> > How can I control it to not write to disk until the segment has reached
> its
> > maximum permitted size (1945 MB?) ? My write traffic is new 
>only (i.e.,
> > it doesnt delete any document) , however I also found following
> infostream
> > logs, which incorrectly say delete=true:
> >
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-887) [   x:filesearch]
> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/update
> > params={commit=false} status=0 QTime=21
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges?
> > numDocsInRam=4434 deletes=true hasTickets:false
> pendingChangesInFullFlush:
> > false
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent:
> infoVersion
> > matches: false; DW changes: true; BD changes: false
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/admin/luke
> > params={show=index=0=json} status=0 QTime=0
> >
> >
> >
> > Thanks
> > Nawab
> >
>


<>

Re: solr 7.0: What causes the segment to flush

[Nawab Zada Asad Iqbal]

I take my yesterday's comment back. I assumed that the file being written
is a segment, however after letting solr run for the night. I see that the
segment is flushed at the expected size:1945MB (so that file which i
observed was still open for writing).
Now, I have two other questions:-

1. Is there a way to not write to disk continuously and only write the file
when segment is flushed?

2. With 6.5: i had ramBufferSizeMB=20G and limiting the threadCount to 12
(since LUCENE-6659
,
there is no configuration for indexing thread count, so I did a local
workaround to limit the number of threads in code); I had very good write
throughput. But with 7.0, I am getting comparable throughput only at
indexing threadcount > 50. What could be wrong ?


Thanks @Erick, I checked the commit settings, both soft and hard commits
are off.




On Tue, Oct 17, 2017 at 3:47 AM, Amrit Sarkar 
wrote:

> >
> > In 7.0, i am finding that the file is written to disk very early on
> > and it is being updated every second or so. Had something changed in 7.0
> > which is causing it?  I tried something similar with solr 6.5 and i was
> > able to get almost a GB size files on disk.
>
>
> Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting
> 20GB segments on 6.5 or less? a GB?
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>
> On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal 
> wrote:
>
> > Hi,
> >
> > I have  tuned  (or tried to tune) my settings to only flush the segment
> > when it has reached its maximum size. At the moment,I am using my
> > application with only a couple of threads (i have limited to one thread
> for
> > analyzing this scenario) and my ramBufferSizeMB=2 (i.e. ~20GB). With
> > this, I assumed that my file sizes on the disk will be at in the order of
> > GB; and no segments will be flushed until the segment's in memory size is
> > 2GB. In 7.0, i am finding that the file is written to disk very early on
> > and it is being updated every second or so. Had something changed in 7.0
> > which is causing it?  I tried something similar with solr 6.5 and i was
> > able to get almost a GB size files on disk.
> >
> > How can I control it to not write to disk until the segment has reached
> its
> > maximum permitted size (1945 MB?) ? My write traffic is 'new only' (i.e.,
> > it doesn't delete any document) , however I also found following
> infostream
> > logs, which incorrectly say 'delete=true':
> >
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-887) [   x:filesearch]
> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/update
> > params={commit=false} status=0 QTime=21
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges?
> > numDocsInRam=4434 deletes=true hasTickets:false
> pendingChangesInFullFlush:
> > false
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent:
> infoVersion
> > matches: false; DW changes: true; BD changes: false
> > Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> > o.a.s.c.S.Request [filesearch]  webapp=/solr path=/admin/luke
> > params={show=index=0=json} status=0 QTime=0
> >
> >
> >
> > Thanks
> > Nawab
> >
>

Re: solr 7.0: What causes the segment to flush

[Erick Erickson]

Segments will also be created whenever, say, time-based commits trip,
i.e. if you have an autoCommit setting to N seconds.

And if the client (SolrJ) submits queries with commitWithin (or, even
worse sends commits).

BTW, I'd set my rambuffer size to 5g as that's the default max size in
tieredMergePolicy and there are some ramifications to having segments
larger than that, namely they must have < 2.5G "live" documents before
being considered for merging. Note, this doesn't matter at all if
there are no deletions.

Best,
Erick

On Tue, Oct 17, 2017 at 3:47 AM, Amrit Sarkar  wrote:
>>
>> In 7.0, i am finding that the file is written to disk very early on
>> and it is being updated every second or so. Had something changed in 7.0
>> which is causing it?  I tried something similar with solr 6.5 and i was
>> able to get almost a GB size files on disk.
>
>
> Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting
> 20GB segments on 6.5 or less? a GB?
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>
> On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal 
> wrote:
>
>> Hi,
>>
>> I have  tuned  (or tried to tune) my settings to only flush the segment
>> when it has reached its maximum size. At the moment,I am using my
>> application with only a couple of threads (i have limited to one thread for
>> analyzing this scenario) and my ramBufferSizeMB=2 (i.e. ~20GB). With
>> this, I assumed that my file sizes on the disk will be at in the order of
>> GB; and no segments will be flushed until the segment's in memory size is
>> 2GB. In 7.0, i am finding that the file is written to disk very early on
>> and it is being updated every second or so. Had something changed in 7.0
>> which is causing it?  I tried something similar with solr 6.5 and i was
>> able to get almost a GB size files on disk.
>>
>> How can I control it to not write to disk until the segment has reached its
>> maximum permitted size (1945 MB?) ? My write traffic is 'new only' (i.e.,
>> it doesn't delete any document) , however I also found following infostream
>> logs, which incorrectly say 'delete=true':
>>
>> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-887) [   x:filesearch]
>> o.a.s.c.S.Request [filesearch]  webapp=/solr path=/update
>> params={commit=false} status=0 QTime=21
>> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges?
>> numDocsInRam=4434 deletes=true hasTickets:false pendingChangesInFullFlush:
>> false
>> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent: infoVersion
>> matches: false; DW changes: true; BD changes: false
>> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
>> o.a.s.c.S.Request [filesearch]  webapp=/solr path=/admin/luke
>> params={show=index=0=json} status=0 QTime=0
>>
>>
>>
>> Thanks
>> Nawab
>>

Re: solr 7.0: What causes the segment to flush

[Amrit Sarkar]

>
> In 7.0, i am finding that the file is written to disk very early on
> and it is being updated every second or so. Had something changed in 7.0
> which is causing it?  I tried something similar with solr 6.5 and i was
> able to get almost a GB size files on disk.


Interesting observation, Nawab, with ramBufferSizeMB=20G, you are getting
20GB segments on 6.5 or less? a GB?

Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2

On Tue, Oct 17, 2017 at 12:48 PM, Nawab Zada Asad Iqbal 
wrote:

> Hi,
>
> I have  tuned  (or tried to tune) my settings to only flush the segment
> when it has reached its maximum size. At the moment,I am using my
> application with only a couple of threads (i have limited to one thread for
> analyzing this scenario) and my ramBufferSizeMB=2 (i.e. ~20GB). With
> this, I assumed that my file sizes on the disk will be at in the order of
> GB; and no segments will be flushed until the segment's in memory size is
> 2GB. In 7.0, i am finding that the file is written to disk very early on
> and it is being updated every second or so. Had something changed in 7.0
> which is causing it?  I tried something similar with solr 6.5 and i was
> able to get almost a GB size files on disk.
>
> How can I control it to not write to disk until the segment has reached its
> maximum permitted size (1945 MB?) ? My write traffic is 'new only' (i.e.,
> it doesn't delete any document) , however I also found following infostream
> logs, which incorrectly say 'delete=true':
>
> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-887) [   x:filesearch]
> o.a.s.c.S.Request [filesearch]  webapp=/solr path=/update
> params={commit=false} status=0 QTime=21
> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> o.a.s.u.LoggingInfoStream [DW][qtp761960786-889]: anyChanges?
> numDocsInRam=4434 deletes=true hasTickets:false pendingChangesInFullFlush:
> false
> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> o.a.s.u.LoggingInfoStream [IW][qtp761960786-889]: nrtIsCurrent: infoVersion
> matches: false; DW changes: true; BD changes: false
> Oct 16, 2017 10:18:29 PM INFO  (qtp761960786-889) [   x:filesearch]
> o.a.s.c.S.Request [filesearch]  webapp=/solr path=/admin/luke
> params={show=index=0=json} status=0 QTime=0
>
>
>
> Thanks
> Nawab
>