My fault for the typo, it slipped in while I wrote the mail.
Each server has 290GB mounted on /opt/solr, contains 2 different replicas
and has a total of 30GB free space.
This is why I wrote 15GB per shard
On Sun, Sep 8, 2013 at 2:56 PM, Erick Erickson wrote:
> How much free disk space do
You're in the wrong room! (Lucene/Java-user) Move the discussion over to the
Solr user list.
-- Jack Krupansky
-Original Message-
From: Erick Erickson
Sent: Sunday, September 08, 2013 7:56 AM
To: java-user
Subject: Re: Expunge deleting using excessive transient disk space
How
How much free disk space do you have when you
try the merge?
Is this a typo?
2
Note name=|
Best
Erick
On Sun, Sep 8, 2013 at 7:26 AM, Manuel Le Normand <
manuel.lenorm...@gmail.com> wrote:
> Hi again,
> In order to delete part of my index I run a delete by query that intends t
disk that can be used (as I have only
15GB per shard).
This procedure failed on a no space left on device exception, although
proper calculations show that these params should cause no usage excess of
the transient free disk space I have.
Looking on the infostream I can see that the first merges
>
>> Any help or hint on possible solutions to fix the above issues is highly
>> appreciated.
>>
>> Thanks.
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Issue-with-disk-space-on-UNIX-tp2676784p2676784.html
>&g
> increasing day by day until the server runs out of memory in a few weeks.
> This happens both on my Windows and Unix servers.
>
> Any help or hint on possible solutions to fix the above issues is highly
> appreciated.
>
> Thanks.
>
> --
> View this mess
hint on possible solutions to fix the above issues is highly
appreciated.
Thanks.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Issue-with-disk-space-on-UNIX-tp2676784p2676784.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com
success we delete the old master and rename the new
> master into it's place.
>
> We're doing disk space checks prior to merge, based on the docs here:
> http://lucene.apache.org/java/2_3_2/api/org/apache/lucene/index/IndexWriter.html#optimize()
>
> but I disabled
master index. On success we delete the old master and rename the new
master into it's place.
We're doing disk space checks prior to merge, based on the docs here:
http://lucene.apache.org/java/2_3_2/api/org/apache/lucene/index/IndexWriter.html#optimize()
but I disabled these to test this ou
On Mon, Jun 7, 2010 at 6:18 AM, Regan Heath
wrote:
>
> That's pretty much exactly what I suspected was happening. I've had the same
> problem myself on another occasion... out of interest is there any way to
> force the file closed without flushing?
No, IndexOutput has no such method. We could
am has handles to these three files
>>> open.
>>>
>>>
>>> Erick Erickson wrote:
>>>>
>>>> What op system and what file system are you using? Is the file system
>>>> local
>>>> or
>>>> networke
three files open.
>>
>>
>> Erick Erickson wrote:
>>>
>>> What op system and what file system are you using? Is the file system
>>> local
>>> or
>>> networked?
>>>
>>> What does it take to remove the files. That is, can yo
That is, can you do it manually
>> after
>> the
>> program shuts down?
>>
>> Best
>> Erick
>>
>
--
View this message in context:
http://lucene.472066.n3.nabble.com/indexWriter-addIndexes-Disk-space-and-open-files-tp841735p875713.html
Sent from the Lucene
geMasterIndex, new StandardAnalyzer(),
>> true);
>> indexWriter.setMaxBufferedDocs(-1);
>> indexWriter.setMaxMergeDocs(2147483647);
>> indexWriter.setMergeFactor(10);
>> indexWriter.setMaxFieldLength(1);
>> indexWriter.addIndexes(indexReaders);
>>
47);
> indexWriter.setMergeFactor(10);
> indexWriter.setMaxFieldLength(10000);
> indexWriter.addIndexes(indexReaders);
> indexWriter.optimize();
>
> This throws an IOException, due to lack of disk space (testing with small
> indexes and a virtual 10mb disk..
> http://www.ltr-data.se/op
ption, due to lack of disk space (testing with small
indexes and a virtual 10mb disk..
http://www.ltr-data.se/opencode.html#ImDisk)
At this point we close all the readers, and the writer and attempt to
cleanup/delete the 'failed' new index directory and files. The problem is
that there are s
You can store the fields in the index itself if you want, without indexing them
(just flag it as stored/unindexed). I believe storing fields should not incur
the "norms" size problem, please correct me if I'm wrong.
Thanks,
Xiaocheng
maureen tanuwidjaja <[EMAIL PROTECTED]> wrote: Ya...I think i
Ya...I think i will store it in the database so that later it could be used in
scoring/ranking for retrieval...:)
Another thing i would like to see is whether the precision or recall will be
much affaected by this...
Regards,
Maureen
Xiaocheng Luan <[EMAIL PROTECTED]> wrote:One side
One side-effect of turning off the norms may be that the scoring/ranking will
be different? Do you need to search by each of these many fields? If not, you
probably don't have to index these fields (but store them for retrieval?).
Just a thought.
Xiaocheng
Michael McCandless <[EMAIL PROTECTED]>
Hi Mike,
How to disable/turn off the norm?is it while indexing?
Thanks,
Maureen
-
Need Mail bonding?
Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users.
"maureen tanuwidjaja" <[EMAIL PROTECTED]> wrote:
> "The only simple workaround I can think of is to set maxMergeDocs to
> keep all segments "small". But then you may have too many segments
> with time. Either that or find a way to reduce the number of unique
> fields that you actually need to
"Michael McCandless" <[EMAIL PROTECTED]> wrote:
> The only simple workaround I can think of is to set maxMergeDocs to
> keep all segments "small". But then you may have too many segments
> with time. Either that or find a way to reduce the number of unique
> fields that you actually need to sto
Oops sorry,mistyping..
I have the searching result in 30 SECONDS to 3 minutes, which is actually
quite unacceptable for the "search engine" I build...Is there any
recommendation on how faster searching could be done?
maureen tanuwidjaja <[EMAIL PROTECTED]> wrote: Hi mike
"The on
Hi mike
"The only simple workaround I can think of is to set maxMergeDocs to
keep all segments "small". But then you may have too many segments
with time. Either that or find a way to reduce the number of unique
fields that you actually need to store."
It is not possible for me to reduce
"maureen tanuwidjaja" <[EMAIL PROTECTED]> wrote:
> "One thing that stands out in your listing is: your norms file
> (_1ke1.nrm) is enormous compared to all other files. Are you indexing
> many tiny docs where each docs has highly variable fields or
> something?"
>
> Ya I also confuse
reen tanuwidjaja" wrote:
> How much actually the disk space needed to optimize the index?The
> explanation given in documentation seems to be very different with the
> practical situation
>
> I have an index file of size 18.6 G and I am going to optimize it.I
> ke
"maureen tanuwidjaja" <[EMAIL PROTECTED]> wrote:
> How much actually the disk space needed to optimize the index?The
> explanation given in documentation seems to be very different with the
> practical situation
>
> I have an index file of size 18.6 G
Dear All
How much actually the disk space needed to optimize the index?The
explanation given in documentation seems to be very different with the
practical situation
I have an index file of size 18.6 G and I am going to optimize it.I keep
this index in mobile Hard Disk with
First, it probably would have been a good thing to start a new thread on
this topic, since it's only vaguely related to disk space ...
That said, sure. Note that there's no requirement in lucene that all
documents in an index have the same fields. Also, there's no reason yo
uble its original size. This results in filling up the disk
space. We have allotted a f/s size of 50G and even that is not
sufficient at times. Is there an optimum vales for the f/s size to
be allotted in such scenario.
Any suggestions would be appreciated.
I believe optimize should use at mos
Harini Raghavan wrote:
Yes I think I got hit IOException. I assumed that the.tmp files are not
required and deleted them manually from the indes directory as they were
more than 10G. Is that ok?
Yes, they are indeed not necessary so deleting them is fine. This
(deleting partially created file
Yes I think I got hit IOException. I assumed that the.tmp files are not
required and deleted them manually from the indes directory as they were
more than 10G. Is that ok?
Michael McCandless wrote:
Harini Raghavan wrote:
Thank you for the response. I don't have readers open on the index,
bu
Harini Raghavan wrote:
Thank you for the response. I don't have readers open on the index, but
while the optimize/merge was running I was searching on the index. Would
that make any difference?
You're welcome! Right, a searcher opens an IndexReader. So this
means you should see peak @ 3X th
ometimes grows to more
than double its original size. This results in filling up the disk
space. We have allotted a f/s size of 50G and even that is not
sufficient at times. Is there an optimum vales for the f/s size to
be allotted in such scenario.
Any suggestions would be appreciated.
I
ere is a Quartz job that is adding
data to the index evey min and around 2 documents get added to
the index every day.When the documents are added and the segments are
merged, the index size increases and sometimes grows to more than
double its original size. This results in filling up the disk sp
On 12/21/06, Michael McCandless <[EMAIL PROTECTED]> wrote:
I *think* it's really max 2X even with compound file (if no readers)?
Because, in IndexWriter.mergeSegments we:
1. Create the newly merged segment in non-compound format (brings us
up to 2X, when it's the last merge).
2. Co
rtz job that is adding data to the index evey
> min and around 2 documents get added to the index every day.When
the
> documents are added and the segments are merged, the index size
> increases and sometimes grows to more than double its original size.
> This results in filling
ing data to the index evey
> min and around 2 documents get added to the index every day.When the
> documents are added and the segments are merged, the index size
> increases and sometimes grows to more than double its original size.
> This results in filling up the disk space. We ha
every day.When the
documents are added and the segments are merged, the index size
increases and sometimes grows to more than double its original size.
This results in filling up the disk space. We have allotted a f/s size
of 50G and even that is not sufficient at times. Is there an optimum
@lucene.apache.org
Subject: Merge Index Filling up Disk Space
Hi All,
I am using lucene 1.9.1 for search functionality in my j2ee application
using JBoss as app server. The lucene index directory size is almost 20G
right now. There is a Quartz job that is adding data to the index evey min
and arou
20 G than your just playing with fire sitting on a 50G
partition. With the price of disk-space these days I would recommend
throwing a lot more storage at your appserver. I would hate to keep
dancing around as I gave it the few more Gig it needed to keep from
crashing.
- Mark
Harini Raghavan
the
documents are added and the segments are merged, the index size
increases and sometimes grows to more than double its original size.
This results in filling up the disk space. We have allotted a f/s size
of 50G and even that is not sufficient at times. Is there an optimum
vales for the f/s
when my lucene index doesn't have yet 300 mb my hard
disk has not already free space, more incredible is that when I turn off
the
process of indexing then the free disk space arise rapidly to 120 Gb. How
could happen this if I doesn't copy the documents to the disk ??? , I
have a
lin
Here is my source code where I convert pdf files to text for indexing, I
got this source code from lucene in action examples and adapted it for my
convenience, I hop you could help me to fix this problem, anyway if you know
another more efficient way to do it please tell me how to:
import java.i
has not already free space, more incredible is that when I turn
off the
process of indexing then the free disk space arise rapidly to 120 Gb. How
could happen this if I doesn't copy the documents to the disk ??? , I
have a
linux machine for the indexing process, I have been thinking that
cou
have yet 300 mb my hard
disk has not already free space, more incredible is that when I turn off the
process of indexing then the free disk space arise rapidly to 120 Gb. How
could happen this if I doesn't copy the documents to the disk ??? , I have a
linux machine for the indexing process, I
i've emptied my index directory but still not much space being freed. Is
there another directory whereby the documents might be stored?
This e-mail has been sent via JARING webmail at http://www.jaring.my
-
47 matches
Mail list logo
