Hi Mike,
I changed my program and now the indexing is better. How ever I have run
into another issue -
I get characters like -
�� - CTA -
in the solr index. I am adding Java beans to solr by the addBean() function.
This seems to be a character encoding issue. Any poi
Hi Mike,
Thanks, I have asked there also, they are investigating, will let you know
if something turns up on that front :)
On Thu, Oct 24, 2013 at 1:30 AM, Michael McCandless <
luc...@mikemccandless.com> wrote:
> Hi Chris,
>
> Sorry, I don't know much about Solr cloud; maybe as on the solr-use
Hi Chris,
Sorry, I don't know much about Solr cloud; maybe as on the solr-user
list, and give details about what went wrong?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Oct 23, 2013 at 11:25 AM, Chris wrote:
> Wow !!! Thanks a lot for the helpfull tips I will implement this in the
Wow !!! Thanks a lot for the helpfull tips I will implement this in the
next two days & report back with my indexing speedI have one more
question...
i tried committing to solr cloud, but then something was not correct
as it would not index after a few documents...
Also, There seems to be som
Indexing 100M web pages really should not take months; if you fix
committing after every row that should make things much faster.
Use multiple index threads, set a highish RAM buffer (~512 MB), use a
local disk not a remote mounted fileserver, ideally an SSD, etc. See
http://wiki.apache.org/lucen
Actually, it contains about 100 million webpages and was built out of a web
index for NLP processing :(
I did the indexing & crawling over one small sized serverand
researching and getting it all to this stage took me this much time...and
now my index is un-usable :(
On Wed, Oct 23, 2013 at
On Wed, Oct 23, 2013 at 10:33 AM, Chris wrote:
> I am not exactly sure if the commit() was run, as i am inserting each row &
> doing a commit right away. My solr will not load the index
I'm confused: if you are doing a commit right away after every row
(which is REALLY bad practice: that's in
t _sa75.nvd _sa8q_Lucene41_0.doc
> > _s91x.fdx _sa75.nvm _sa8q_Lucene41_0.pos
> > _s91x.fnm _sa75.si _sa8q_Lucene41_0.tim
> > _s91x_i.del _sa75.tvd _sa8q_Lucene41_0.tip
> > _s91x_Luc
8q.tvx
> _s91x.nvm _sa7s_Lucene41_0.doc segments.gen
> _s91x.si _sa7s_Lucene41_0.pos
> _s91x.tvd _sa7s_Lucene41_0.tim
>
>
>
> On Wed, Oct 23, 2013 at 7:07 PM, Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>>
t; > I am running solr 4.4 & one of my collections seems to have a corrupted
> > index...
> >
> > I tried doing -
> > java -cp lucene-core-4.4.0.jar -ea:org.apache.lucene...
> > org.apache.lucene.index.CheckIndex /solr2/example/solr/w1/data/index/
How did this corruption happen?
If you "ls" your index directory, is there any segments_N file?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Oct 23, 2013 at 9:01 AM, Chris wrote:
> Hi,
>
> I am running solr 4.4 & one of my collections seems to have a cor
Hi,
I am running solr 4.4 & one of my collections seems to have a corrupted
index...
I tried doing -
java -cp lucene-core-4.4.0.jar -ea:org.apache.lucene...
org.apache.lucene.index.CheckIndex /solr2/example/solr/w1/data/index/ -fix
But it didnt help...gives -
ERROR: could not read any segm
Hi
I have a large Index with a open IndexWriter.some one unknowlingly deleted
some files in the index and corrupted the index.
Once i get IndexCorruptedException,I will delete the entire Index Folder
and close my indexwriter also and Restart indexing process again.
In that time,I am Still Getti
unusally small it will be detected as
corrupted.
Thanks again for your help !
--
View this message in context:
http://lucene.472066.n3.nabble.com/Detect-a-corrupted-index-tp4080747p4081114.html
Sent from the Lucene - Java Users mailing list arc
new File(path));
}
///
Best regards, Lisheng
-Original Message-
From: ABlaise [mailto:abla...@hotpads.com]
Sent: Friday, July 26, 2013 7:33 PM
To: java-user@lucene.apache.org
Subject: Detect a corrupted index
Hi everyone !
So I am working on a Lucene index
ratch doesn't take too
long.
Any idea or hint on how I should proceed ? I searched for related topics but
they all seemed old and don't apply to Lucene 4.3.1
Thanks in advance
Adrien
--
View this message in context:
http://lucene.472066.n3.nabble.com/Detect-a-corrupted-index-tp4080747
Hello!
Just one thing - backup your index first, just in case.
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hello!
> Try using CheckIndex -
> http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/api/all/org/apache/lucene/index/CheckIndex.html
Hello!
Try using CheckIndex -
http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/api/all/org/apache/lucene/index/CheckIndex.html
--
Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hi,
> I have an index, for which I am missing at least 1 file after
Hi,
I have an index, for which I am missing at least 1 file after hitting a
disk full situation.
is there any way I could bypass the error I get when trying to open the
index, to salvage as many docs as I can from the other files?
thanks,
vince
java.io.FileNotFoundException: D:\_2c9kgw.cfs (T
: i use SimpleFSLockFactory and optimized opened IndexWriter, but i use api to
: forcefully removing the lock file.
: this missing file, just occurs in sometimes not always??
forcibly removing hte lock file isn't something that should be done in a
typical situation ... if the lock file is there,
egmentReader.java:181)
>>>>> 16:20:57,565 INFO [STDOUT] : 140 at
>>>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:167)
>>>>> 16:20:57,565 INFO [STDOUT] : 140 at
>>>>> org.apache.lucene.index.SegmentReader.
ember, 2008 8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
It's perfectly fine to have a reader open on an index, while an
IndexWriter runs optimize.
Which version of Lucene are you using? And which OS & filesystem?
Mike
Chaula Ganatra wrote:
It
gt; 16:20:57,565 INFO [STDOUT] : 140 at
>>> org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:1867)
>>> 16:20:57,565 INFO [STDOUT] : 140 at
>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:1231)
>>>
>>>
&g
r, 2008 8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
It's perfectly fine to have a reader open on an index, while an
IndexWriter runs optimize.
Which version of Lucene are you using? And which OS & filesystem?
Mike
Chaula Ganatra wrote:
It was
lso getting the same error.
Can anyone please help me out? It is occurring on live environment.
Regards,
Chaula
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
chael McCandless [mailto:[EMAIL PROTECTED]
> Sent: 26 September, 2008 8:00 PM
> To: java-user@lucene.apache.org
> Subject: Re: How to restore corrupted index
>
>
> It's perfectly fine to have a reader open on an index, while an
> IndexWriter runs optimize.
>
&g
environment.
Regards,
Chaula
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
It's perfectly fine to have a reader open on an index, while an
IndexW
Chaula Ganatra wrote:
We are using 2.2.0 and at least for our coming release we will be
using
the same, we will upgrade to the latest one after that. So is it a bug
in 2.2.0.
I can't tell if this is a bug yet... I need more details, like the
stack traces, but
also more details about how y
, 2008 9:50 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
OK. I really need to see those stack traces to better understand this
issue.
Also, does the issue still happen on 2.3, or 2.4 RC2?
Mike
Chaula Ganatra wrote:
> Lucene 2.2.0, windows XP
>
>
id not close so gave
exception
in writer.optimise()
Chaula
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 7:17 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
Can you post the full stack trace in both cases?
Mike
ze.
>>>
>>> Which version of Lucene are you using? And which OS & filesystem?
>>>
>>> Mike
>>>
>>> Chaula Ganatra wrote:
>>>
>>>> It was the Reader on same index, which I did not close so gave exception
>>>> i
8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
It's perfectly fine to have a reader open on an index, while an
IndexWriter runs optimize.
Which version of Lucene are you using? And which OS & filesystem?
Mike
Chaula Ganatra wrote:
It was the
dless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 7:17 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
Can you post the full stack trace in both cases?
Mike
Chaula Ganatra wrote:
I found one case when such multiple files are remained, when we
call
writer.
t; Sent: 26 September, 2008 7:17 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: How to restore corrupted index
>>
>>
>> Can you post the full stack trace in both cases?
>>
>> Mike
>>
>> Chaula Ganatra wrote:
>>
>>> I found
Lucene 2.2.0, windows XP
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 8:00 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
It's perfectly fine to have a reader open on an index, while an
IndexWriter
)
Chaula
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 7:17 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
Can you post the full stack trace in both cases?
Mike
Chaula Ganatra wrote:
I found one case
It was the Reader on same index, which I did not close so gave exception
in writer.optimise()
Chaula
-Original Message-
From: Michael McCandless [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 7:17 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
Can
calling
writer.addDocument it throws java.lang.NegativeArraySizeException
Regards,
Chaula
-Original Message-
From: Grant Ingersoll [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 6:02 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
There is the CheckIndex tool
,
Chaula
-Original Message-
From: Grant Ingersoll [mailto:[EMAIL PROTECTED]
Sent: 26 September, 2008 6:02 PM
To: java-user@lucene.apache.org
Subject: Re: How to restore corrupted index
There is the CheckIndex tool included in the distribution for checking/
fixing bad indexes, but it can&#
There is the CheckIndex tool included in the distribution for checking/
fixing bad indexes, but it can't solve everything.
The bigger question is why it is happening to begin with. Can you
describe your indexing process? How do you know the index is actually
corrupted? Are you seeing exce
You say that there are multiple files, but you don't say if the index
still works. Does it? If using the index gives you unexpected results,
can you tell us about what the failure modes are?
Best
Erick
On Fri, Sep 26, 2008 at 6:49 AM, Chaula Ganatra <[EMAIL PROTECTED]> wrote:
> We have an applic
We have an application in which index will be updated frequently.
During development time, found that index files gets corrupted, i.e.
more than one cfs files,some other extension files e.g. frq, fnm, nrm
Remains there in index directory.
Is there any way that such issue does not occur at all o
o java-user, so somebody
else can put it there.
Otis
- Original Message
From: Olivier Jaquemet <[EMAIL PROTECTED]>
To: Lucene Java User ML
Sent: Wednesday, March 22, 2006 10:08:28 AM
Subject: FileNotFoundException: Corrupted Index?
Hi all,
We are using the last version o
o somebody
else can put it there.
Otis
- Original Message
From: Olivier Jaquemet <[EMAIL PROTECTED]>
To: Lucene Java User ML
Sent: Wednesday, March 22, 2006 10:08:28 AM
Subject: FileNotFoundException: Corrupted Index?
Hi all,
We are using the last version of lucene (1.9.1), and som
available in this case being to completely remove and
recreate the index.
I have the "corrupted" index available for testing should you need it.
Apparently this corruption occurs if the JVM has crashed or was shutdown
too violently (kill -9)
I was wondering how a corruption of a lucene i
Bill Tschumy wrote:
So, did this happen because he copied the data while in an inconsistent
state? I'm a bit surprised that an inconsistent index is ever left on
disk (except for temporarily while something is being written). Would
this happen if there was a Writer that was not closed?
An inde
Daniel Naber wrote:
Yes, the *.cfs shows that this is a compound index which has *.fnm files
only when it's being modified.
When creating a compound segment, a "segments" file is never written
that refers to the segment until the .cfs file is created and the .fnm
files are removed.
The real pro
Daniel,
Thanks for responding on this thread. I doubt the copy was made while
the index was being updated and I don't see any indication of a crash.
Just for my clarification, if I update the index, but don't close the
IndexWriter (because I may need it again soon), can the index on disk
be le
On Friday 08 April 2005 23:51, Bill Tschumy wrote:
> Would
> this happen if there was a Writer that was not closed?
Either the copy was done while the index was being updated, or the previous
index update didn't finish (e.g. because it crashed before the index was
closed).
Regards
Daniel
--
So, did this happen because he copied the data while in an inconsistent
state? I'm a bit surprised that an inconsistent index is ever left on
disk (except for temporarily while something is being written). Would
this happen if there was a Writer that was not closed?
On Apr 8, 2005, at 1:22 PM
On Friday 08 April 2005 19:26, Bill Tschumy wrote:
> The only thought I had was that he copied the data while the app was Â
> still running and perhaps it was in an inconsistent state.
Yes, the *.cfs shows that this is a compound index which has *.fnm files
only when it's being modified. You're
On Friday 08 April 2005 19:26, Bill Tschumy wrote:
> The only thought I had was that he copied the data while the app was Â
> still running and perhaps it was in an inconsistent state.
Yes, the *.cfs shows that this is a compound index which has *.fnm files
only when it's being modified. You're
I had a customer report a corrupted Lucene index. He had copied the
index to backup storage, reformatted his drive, and then restored the
data. After that Lucene has trouble opening the index.
Here is the backtrace:
java.io.FileNotFoundException: /Users/bill/Desktop/IndexData/_v.fnm (No
su
53 matches
Mail list logo
