Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Shawn Heisey]

On 11/28/2016 11:06 AM, Walter Underwood wrote:
> Worst case:
> 1. Disable merging.
> 2. Delete all the documents.
> 3. Add all the documents.
> 4. Enable merging.
>
> After step 3, you have two copies of everything, one deleted copy and one new 
> copy.
> The merge makes a third copy.

Just getting around to replying to this REALLY old thread.

What you've described doesn't really triple the size of the index at the
optimize/forceMerge step.  While it's true that the index is temporarily
three times it's *final* size, it is not three times the pre-optimize
size -- in fact, it would only get 50 percent larger.

Does this mean that the recommendation saying "need 3x the space for
normal operation" is not in fact true?  The Lucene folks seem to be
pretty adamant in their assertion that a merge to one segment can triple
the index size, although I've never seen it actually happen.  Disabling
and re-enabling merging is not what I would call "normal."

Thanks,
Shawn

Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Walter Underwood]

> On Nov 28, 2016, at 9:38 AM, Shawn Heisey  wrote:
> 
> […] Typically
> a merge or optimize will only require double the space, but there are
> certain worst-case scenarios where it can require triple.  I do not know
> what causes the worst-case situation.

Worst case:

1. Disable merging.
2. Delete all the documents.
3. Add all the documents.
4. Enable merging.

After step 3, you have two copies of everything, one deleted copy and one new 
copy.
The merge makes a third copy.

This can happen with any search engine that uses the same kind of segment
management as Lucene. We learned about this with Ultraseek, back in the
mid-1990’s.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Michael Joyner]

On 11/28/2016 12:26 PM, Erick Erickson wrote:

Well, such checks could be put in, but they don't get past the basic problem.



And all this masks your real problem; you didn't have enough disk
space to optimize in the first place. Even during regular indexing w/o
optimizing, Lucene segment merging can always theoretically merge all
your segments at once. Therefore you always need at _least_ as much
free space on your disks as all your indexes occupy to be sure you
won't hit a disk-full problem. The rest would be band-aids. Although I
suppose refusing to even start if there wasn't enough free disk space
isn't a bad idea, it's not foolproof though


If such a "warning" feature is added, it's not "foolproof" that I would 
expect, and I wouldn't expect it be able to "predict" usage caused by 
events happening after it passes a basic initial check. I am just 
thinking a basic up-front check that indicates "it just ain't happening" 
might be useful.


So.. how does one handle needing all this "free space" between major 
index updates when one gets charged by the GB for allocated space 
without regard to actual storage usage?

Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Shawn Heisey]

On 11/28/2016 9:39 AM, Michael Joyner wrote:
> I'm running out of spacing when trying to restart nodes to get a
> cluster back up fully operational where a node ran out of space during
> an optimize.
>
> It appears to be trying to do a full sync from another node, but
> doesn't take care to check available space before starting downloads
> and doesn't delete the out of date segment files before attempting to
> do the full sync.

If you've run out of space during an optimize, then your Solr install
doesn't have enough disk space for proper operation.  The recommendation
is to have enough disk space to store all your index data three times --
free space should be double the size of all your index data.  Typically
a merge or optimize will only require double the space, but there are
certain worst-case scenarios where it can require triple.  I do not know
what causes the worst-case situation.  This is a Lucene requirement, and
Solr is based on Lucene.

The replication feature, which is how SolrCloud accomplishes index
recovery, assumes that the existing index must remain online until the
new index is fully transferred and available, at which time it will
become the live index, and the previous one can be deleted.  This
feature existed long before SolrCloud did.  Standalone mode will not be
disappearing anytime soon, so this assumption must remain.  Writing code
to decide when the existing index doesn't need to be kept would be
somewhat difficult and potentially very fragile.  This doesn't mean we
won't do it, but I think that's why it hasn't already been done.

Also, we still have that general disk space recommendation already
mentioned.  If that recommendation is followed, you're not going to run
out of disk space due to index recovery.

> It seems to know what size the segments are before they are
> transferred, is there a reason a basic disk space check isn't done for
> the target partition with an immediate abort done if the destination's
> space looks like it would go negative before attempting sync? Is this
> something that can be enabled in the master solrconfig.xml file? This
> would be a lot more useful (IMHO) than waiting for a full sync to
> complete only to run out of space after several hundred gigs of data
> is transferred with automatic cluster recovery failing as a result.

Remembering that the replication feature is NOT limited to use by
SolrCloud ... this is not a bad idea.  Because the replication handler
knows what files must be transferred before an index fetch takes place,
it can calculate how much disk space is required, and could return an
error response and ignore the command.  The way that SolrCloud uses
replication may not work with this, though.  SolrCloud replication may
work differently than the automated replication that can be set up in
standalone mode.  I am not sure whether it handles individual files, or
simply requests an index fetch.

But, at the risk of repeating myself ... running with so little free
disk space is not recommended.  The entire problem is avoided by
following recommendations.

Thanks,
Shawn

Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Michael Joyner]

We've being trying to run at 40% estimated usage when optimized, but are 
doing a large amount of index updates ... 40% usage in this scenario 
seems to be too high...



On 11/28/2016 12:26 PM, Erick Erickson wrote:

Well, such checks could be put in, but they don't get past the basic problem.

bq: If the segments are out of date and we are pulling from another
node before coming "online" why aren't the old segments deleted?

because you run the risk of losing _all_ your data and having nothing
at all. The process is
1> pull all the segments down
2> rewrite the segments file

Until <2>, you can still use your old index. Also consider a full
synch in master/slave mode. I optimize on the master and Solr then
detects that it'll be a full sync anddeletes the entire active
index.

bq: Is this something that can be enabled in the master solrconfig.xml file?
no

bq: ...is there a reason a basic disk space check isn't done 
That would not be very robust. Consider the index is 1G and I have
1.5G of free space. Now replication makes the check and starts.
However, during that time segments are merged consuming .75G. Boom,
disk full again.

Additionally, any checks would be per core. What if 10 cores start
replication as above at once? Which would absolutely happen if you
have 10 replicas for the same shard in one JVM...

And all this masks your real problem; you didn't have enough disk
space to optimize in the first place. Even during regular indexing w/o
optimizing, Lucene segment merging can always theoretically merge all
your segments at once. Therefore you always need at _least_ as much
free space on your disks as all your indexes occupy to be sure you
won't hit a disk-full problem. The rest would be band-aids. Although I
suppose refusing to even start if there wasn't enough free disk space
isn't a bad idea, it's not foolproof though

Best,
Erick


On Mon, Nov 28, 2016 at 8:39 AM, Michael Joyner  wrote:

Hello all,

I'm running out of spacing when trying to restart nodes to get a cluster
back up fully operational where a node ran out of space during an optimize.

It appears to be trying to do a full sync from another node, but doesn't
take care to check available space before starting downloads and doesn't
delete the out of date segment files before attempting to do the full sync.

If the segments are out of date and we are pulling from another node before
coming "online" why aren't the old segments deleted? Is this something that
can be enabled in the master solrconfig.xml file?

It seems to know what size the segments are before they are transferred, is
there a reason a basic disk space check isn't done for the target partition
with an immediate abort done if the destination's space looks like it would
go negative before attempting sync? Is this something that can be enabled in
the master solrconfig.xml file? This would be a lot more useful (IMHO) than
waiting for a full sync to complete only to run out of space after several
hundred gigs of data is transferred with automatic cluster recovery failing
as a result.

This happens when doing a 'sudo service solr restart'

(Workaround, shutdown offending node, manually delete segment index folders
and tlog files, start node)

Exception:

WARN  - 2016-11-28 16:15:16.291;
org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching file:
_2f6i.cfs (downloaded 2317352960 of 5257809205 bytes)
java.io.IOException: No space left on device
 at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
 at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
 at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
 at sun.nio.ch.IOUtil.write(IOUtil.java:65)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
 at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
 at java.nio.channels.Channels.writeFully(Channels.java:101)
 at java.nio.channels.Channels.access$000(Channels.java:61)
 at java.nio.channels.Channels$1.write(Channels.java:174)
 at
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419)
 at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
 at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
 at
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
 at
org.apache.solr.handler.IndexFetcher$DirectoryFile.write(IndexFetcher.java:1634)
 at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1491)
 at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1429)
 at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:855)
 at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:434)
 at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
 at

Re: Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Erick Erickson]

Well, such checks could be put in, but they don't get past the basic problem.

bq: If the segments are out of date and we are pulling from another
node before coming "online" why aren't the old segments deleted?

because you run the risk of losing _all_ your data and having nothing
at all. The process is
1> pull all the segments down
2> rewrite the segments file

Until <2>, you can still use your old index. Also consider a full
synch in master/slave mode. I optimize on the master and Solr then
detects that it'll be a full sync anddeletes the entire active
index.

bq: Is this something that can be enabled in the master solrconfig.xml file?
no

bq: ...is there a reason a basic disk space check isn't done 
That would not be very robust. Consider the index is 1G and I have
1.5G of free space. Now replication makes the check and starts.
However, during that time segments are merged consuming .75G. Boom,
disk full again.

Additionally, any checks would be per core. What if 10 cores start
replication as above at once? Which would absolutely happen if you
have 10 replicas for the same shard in one JVM...

And all this masks your real problem; you didn't have enough disk
space to optimize in the first place. Even during regular indexing w/o
optimizing, Lucene segment merging can always theoretically merge all
your segments at once. Therefore you always need at _least_ as much
free space on your disks as all your indexes occupy to be sure you
won't hit a disk-full problem. The rest would be band-aids. Although I
suppose refusing to even start if there wasn't enough free disk space
isn't a bad idea, it's not foolproof though

Best,
Erick


On Mon, Nov 28, 2016 at 8:39 AM, Michael Joyner  wrote:
> Hello all,
>
> I'm running out of spacing when trying to restart nodes to get a cluster
> back up fully operational where a node ran out of space during an optimize.
>
> It appears to be trying to do a full sync from another node, but doesn't
> take care to check available space before starting downloads and doesn't
> delete the out of date segment files before attempting to do the full sync.
>
> If the segments are out of date and we are pulling from another node before
> coming "online" why aren't the old segments deleted? Is this something that
> can be enabled in the master solrconfig.xml file?
>
> It seems to know what size the segments are before they are transferred, is
> there a reason a basic disk space check isn't done for the target partition
> with an immediate abort done if the destination's space looks like it would
> go negative before attempting sync? Is this something that can be enabled in
> the master solrconfig.xml file? This would be a lot more useful (IMHO) than
> waiting for a full sync to complete only to run out of space after several
> hundred gigs of data is transferred with automatic cluster recovery failing
> as a result.
>
> This happens when doing a 'sudo service solr restart'
>
> (Workaround, shutdown offending node, manually delete segment index folders
> and tlog files, start node)
>
> Exception:
>
> WARN  - 2016-11-28 16:15:16.291;
> org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching file:
> _2f6i.cfs (downloaded 2317352960 of 5257809205 bytes)
> java.io.IOException: No space left on device
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
> at java.nio.channels.Channels.writeFully(Channels.java:101)
> at java.nio.channels.Channels.access$000(Channels.java:61)
> at java.nio.channels.Channels$1.write(Channels.java:174)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419)
> at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at
> org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
> at
> org.apache.solr.handler.IndexFetcher$DirectoryFile.write(IndexFetcher.java:1634)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1491)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1429)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:855)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:434)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:397)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
> at
> 

Failure when trying to full sync, out of space ? Doesn't delete old segments before full sync?

[Michael Joyner]

Hello all,

I'm running out of spacing when trying to restart nodes to get a cluster 
back up fully operational where a node ran out of space during an optimize.


It appears to be trying to do a full sync from another node, but doesn't 
take care to check available space before starting downloads and doesn't 
delete the out of date segment files before attempting to do the full sync.


If the segments are out of date and we are pulling from another node 
before coming "online" why aren't the old segments deleted? Is this 
something that can be enabled in the master solrconfig.xml file?


It seems to know what size the segments are before they are transferred, 
is there a reason a basic disk space check isn't done for the target 
partition with an immediate abort done if the destination's space looks 
like it would go negative before attempting sync? Is this something that 
can be enabled in the master solrconfig.xml file? This would be a lot 
more useful (IMHO) than waiting for a full sync to complete only to run 
out of space after several hundred gigs of data is transferred with 
automatic cluster recovery failing as a result.


This happens when doing a 'sudo service solr restart'

(Workaround, shutdown offending node, manually delete segment index 
folders and tlog files, start node)


Exception:

WARN  - 2016-11-28 16:15:16.291; 
org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching 
file: _2f6i.cfs (downloaded 2317352960 of 5257809205 bytes)

java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at 
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419)

at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at 
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
at 
org.apache.solr.handler.IndexFetcher$DirectoryFile.write(IndexFetcher.java:1634)
at 
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1491)
at 
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1429)
at 
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:855)
at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:434)
at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:397)
at 
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at 
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

-Mike