[jira] [Commented] (LUCENE-5050) CompressingStoredFieldsReader should close the index file as soon as it has been read
[
https://issues.apache.org/jira/browse/LUCENE-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680560#comment-13680560
]
Commit Tag Bot commented on LUCENE-5050:
[branch_4x commit] jpountz
http://svn.apache.org/viewvc?view=revisionrevision=1491909
LUCENE-5050: Close the stored fields and term vectors index files eagerly
(merged from r1491889).
CompressingStoredFieldsReader should close the index file as soon as it has
been read
-
Key: LUCENE-5050
URL: https://issues.apache.org/jira/browse/LUCENE-5050
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.3.1
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Fix For: 5.0, 4.4
Attachments: LUCENE-5050.patch
Although CompressingStoredFieldsReader loads the stored fields index into
memory, it only closes the index file in {{close()}}. Closing at the end of
the constructor should help save file descriptors.
The same idea applies to CompressingTermVectorsReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-5050) CompressingStoredFieldsReader should close the index file as soon as it has been read
[
https://issues.apache.org/jira/browse/LUCENE-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand resolved LUCENE-5050.
--
Resolution: Fixed
Thanks Robert.
CompressingStoredFieldsReader should close the index file as soon as it has
been read
-
Key: LUCENE-5050
URL: https://issues.apache.org/jira/browse/LUCENE-5050
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.3.1
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Fix For: 5.0, 4.4
Attachments: LUCENE-5050.patch
Although CompressingStoredFieldsReader loads the stored fields index into
memory, it only closes the index file in {{close()}}. Closing at the end of
the constructor should help save file descriptors.
The same idea applies to CompressingTermVectorsReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-5050) CompressingStoredFieldsReader should close the index file as soon as it has been read
Adrien Grand created LUCENE-5050:
Summary: CompressingStoredFieldsReader should close the index file
as soon as it has been read
Key: LUCENE-5050
URL: https://issues.apache.org/jira/browse/LUCENE-5050
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.3.1
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Fix For: 5.0, 4.4
Although CompressingStoredFieldsReader loads the stored fields index into
memory, it only closes the index file in {{close()}}. Closing at the end of the
constructor should help save file descriptors.
The same idea applies to CompressingTermVectorsReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5050) CompressingStoredFieldsReader should close the index file as soon as it has been read
[
https://issues.apache.org/jira/browse/LUCENE-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-5050:
-
Attachment: LUCENE-5050.patch
Here is a patch which closes the index file in the constructor for both
CompressingStoredFieldsReader and CompressingTermVectorsReader.
CompressingStoredFieldsReader should close the index file as soon as it has
been read
-
Key: LUCENE-5050
URL: https://issues.apache.org/jira/browse/LUCENE-5050
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.3.1
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Fix For: 5.0, 4.4
Attachments: LUCENE-5050.patch
Although CompressingStoredFieldsReader loads the stored fields index into
memory, it only closes the index file in {{close()}}. Closing at the end of
the constructor should help save file descriptors.
The same idea applies to CompressingTermVectorsReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5050) CompressingStoredFieldsReader should close the index file as soon as it has been read
[
https://issues.apache.org/jira/browse/LUCENE-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679785#comment-13679785
]
Robert Muir commented on LUCENE-5050:
-
+1
CompressingStoredFieldsReader should close the index file as soon as it has
been read
-
Key: LUCENE-5050
URL: https://issues.apache.org/jira/browse/LUCENE-5050
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.3.1
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Fix For: 5.0, 4.4
Attachments: LUCENE-5050.patch
Although CompressingStoredFieldsReader loads the stored fields index into
memory, it only closes the index file in {{close()}}. Closing at the end of
the constructor should help save file descriptors.
The same idea applies to CompressingTermVectorsReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [Heads-up] Index file format change on trunk
Hi, I just modified the Lucene 4.1 stored fields format again[1]. You should reindex. [1] https://issues.apache.org/jira/browse/LUCENE-4558 -- Adrien
[Heads-up] Index file format change on trunk
Hi, I just committed LUCENE-4509 [1] which changes the default StoredFieldsFormat on trunk (backport to come on branch 4.x in a few minutes). You should reindex. [1] https://issues.apache.org/jira/browse/LUCENE-4509 -- Adrien
Re: [Heads-up] Index file format change on trunk
Hmmm... does this mean that 4.1 would REQUIRE reindexing?!?! If so, that would strongly argue for there to be a 4.0.1 release that has both bug fixes and the incremental feature additions of the 4x branch but DOESN’T require a reindex. Who made this decision that it was A-OK for 4.1 to require a reindex? I don’t recall seeing any discussion here about it. -- Jack Krupansky From: Adrien Grand Sent: Tuesday, November 13, 2012 7:00 AM To: dev@lucene.apache.org Subject: [Heads-up] Index file format change on trunk Hi, I just committed LUCENE-4509 [1] which changes the default StoredFieldsFormat on trunk (backport to come on branch 4.x in a few minutes). You should reindex. [1] https://issues.apache.org/jira/browse/LUCENE-4509 -- Adrien
Re: [Heads-up] Index file format change on trunk
Hi Jack, On Tue, Nov 13, 2012 at 4:06 PM, Jack Krupansky j...@basetechnology.comwrote: Hmmm... does this mean that 4.1 would REQUIRE reindexing? No it doesn't. This is a notice for people who would already use the (unreleased) Lucene41Codec for some test indexes to tell them that they should reindex if they want their code to keep working with the latest trunk revision. -- Adrien
Re: [Heads-up] Index file format change on trunk
Thank you for that clarification! -- Jack Krupansky From: Adrien Grand Sent: Tuesday, November 13, 2012 7:10 AM To: dev@lucene.apache.org Subject: Re: [Heads-up] Index file format change on trunk Hi Jack, On Tue, Nov 13, 2012 at 4:06 PM, Jack Krupansky j...@basetechnology.com wrote: Hmmm... does this mean that 4.1 would REQUIRE reindexing? No it doesn't. This is a notice for people who would already use the (unreleased) Lucene41Codec for some test indexes to tell them that they should reindex if they want their code to keep working with the latest trunk revision. -- Adrien
[jira] [Commented] (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425593#comment-13425593 ] Faizan Aziz commented on SOLR-1822: --- I am getting the same error for solr 3.5.0. On close analysis of SnapPuller.java I notice that the fix mentioned in the attachment is not done SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Assignee: Mark Miller Priority: Critical Fix For: 3.1, 4.0-ALPHA Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425600#comment-13425600 ] Faizan Aziz commented on SOLR-1822: --- ok The fix by kafka0102 is there so I guess the patch is not needed However I am still getting the error SEVERE: Unable to copy index file from: /opt/solr/multicore/1011/data/index.20120731065949/_9x.tis to: /mnt/multicore/1011/data/index.20120731050740/_9x.tis Note: /opt/solr/multicore is a symlink to /mnt/multicore SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Assignee: Mark Miller Priority: Critical Fix For: 3.1, 4.0-ALPHA Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425605#comment-13425605 ] Faizan Aziz commented on SOLR-1822: --- Complete logs: Jul 31, 2012 6:59:49 AM org.apache.solr.handler.SnapPuller copyAFile SEVERE: Unable to move index file from: /opt/solr/multicore/1011/data/index.20120731065949/_9x.tis to: /mnt/multicore/1011/data/index.20120731050740/_9x.tisTrying to do a copy Jul 31, 2012 6:59:49 AM org.apache.solr.handler.SnapPuller copyAFile SEVERE: Unable to copy index file from: /opt/solr/multicore/1011/data/index.20120731065949/_9x.tis to: /mnt/multicore/1011/data/index.20120731050740/_9x.tis java.io.FileNotFoundException: /mnt/multicore/1011/data/index.20120731050740/_9x.tis (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:209) at java.io.FileOutputStream.init(FileOutputStream.java:160) at org.apache.solr.common.util.FileUtils.copyFile(FileUtils.java:48) at org.apache.solr.handler.SnapPuller.copyAFile(SnapPuller.java:585) at org.apache.solr.handler.SnapPuller.copyIndexFiles(SnapPuller.java:621) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:317) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268) at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:149) SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Assignee: Mark Miller Priority: Critical Fix For: 3.1, 4.0-ALPHA Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[HEADS-UP]: Index File Format Change on Trunk
Hey folks, I just committed LUCENE-4051 [1] (Revision 1341768) which changes the file format of DocValues, Norms (DocValues), StoredFields TermVectors incompatible to previous revisions. If you are using trunk indices you must re-index before updating to the latest trunk sources. If you are using Lucene 3.x or below you can safely ignore this message. happy indexing, simon [1] https://issues.apache.org/jira/browse/LUCENE-4051 - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Heads Up - Index File Format Change on Trunk
Folks, I just committed LUCENE-3628 [1] which cuts over Norms to DocVaues. This is an index file format change and if you are using trunk you need to reindex before updating. happy indexing :) simon [1] https://issues.apache.org/jira/browse/LUCENE-3628 - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
how to customize lucene index file
Hi, good morning everyone. I would like to ask one question about lucene index file. I want to know whether I could customize this index file. I know this index file is created by lucene in the backgroud from documents directory. And all of words and their frequence is output in the index file. In fact I want to calculate the weight of each word and just output some words which weight value is high in the index file. According to the index creation method as below *** ... IndexWriter writer = new IndexWriter(index, new JapaneseAnalyzer(), create); ... Document doc = new Document(); doc.add(Field.Text(contents,reader)); ... writer.addDocument(doc); *** It seems impossible. But I realy want to is there some method to realize it in the lucene ? In fact I want to use lucene not for retrival but for text clustering. Best regards. Yali Hu - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Heads Up - Index File Format Change on Trunk
Simon can you also email java-user@ and solr-user@? Seems good to over-communicate when trunk index format changes... Mike McCandless http://blog.mikemccandless.com On Fri, Jun 10, 2011 at 10:01 AM, Simon Willnauer simon.willna...@googlemail.com wrote: Hey folks, I just committed LUCENE-3108 (Landing DocValues on Trunk) which adds a byte to FieldInfo. If you are running on trunk you must / should re-index any trunk indexes once you update to the latest trunk. its likely if you open up old trunk (4.0) indexes, you will get an exception related to Read Past EOF. Simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Heads Up - Index File Format Change on Trunk
On Fri, Jun 10, 2011 at 6:22 PM, Michael McCandless luc...@mikemccandless.com wrote: Simon can you also email java-user@ and solr-user@? Seems good to over-communicate when trunk index format changes... good point, done! Mike McCandless http://blog.mikemccandless.com On Fri, Jun 10, 2011 at 10:01 AM, Simon Willnauer simon.willna...@googlemail.com wrote: Hey folks, I just committed LUCENE-3108 (Landing DocValues on Trunk) which adds a byte to FieldInfo. If you are running on trunk you must / should re-index any trunk indexes once you update to the latest trunk. its likely if you open up old trunk (4.0) indexes, you will get an exception related to Read Past EOF. Simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Index File
Hi All, I have created Index folder, and i tried to open .CFS,.CFX,.GEN, segments file with notpad . but they are unreadable. i want to see their structure for my sample directory which i have passed to Indexfiles . ihave red this http://lucene.apache.org/java/3_0_3/fileformats.html and i know what Index should has. is there any way to see created index? thanks in advance . Peace. -Soheila D.
Re: Index File
Hi, 2011/3/21 soheila dehghanzadeh sally...@gmail.com: Hi All, I have created Index folder, and i tried to open .CFS,.CFX,.GEN, segments file with notpad . but they are unreadable. i want to see their structure for my sample directory which i have passed to Indexfiles . ihave red this http://lucene.apache.org/java/3_0_3/fileformats.html and i know what Index should has. is there any way to see created index? yes, use the force :) http://code.google.com/p/luke/ Regards, Sanne thanks in advance . Peace. -Soheila D. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Resolved: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller resolved SOLR-1822. --- Resolution: Duplicate Fix Version/s: (was: Next) 4.0 3.1 Assignee: Mark Miller SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Assignee: Mark Miller Priority: Critical Fix For: 3.1, 4.0 Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Index File Format Change - heads up (late)
hey there, a couple of days ago the trunk index format changed with LUCENE-2881 you should re-index any trunk indexes. its likely if you open up old trunk indexes, you will get an exception related to Read Past EOF. Sorry for the delay! simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12934715#action_12934715 ] kafka0102 commented on SOLR-1822: - I found the problem's reason.It's tmpDir's name same to indexDir's name. tmpIdxDirName's format is SnapShooter.DATE_FMT = MMddhhmmss. If I have a indexDirName index.20101122031000 because of a full index repl at 2010-11-22 03:10:00. And slave's repl duration is 5 minitues. So at 2010-11-22 15:10:00, the slave made indexDirName and indexDirName both index.20101122031000, and finnaly will drop the index.After that,replication would generate the above error again and again. SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: Next Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Issue Comment Edited: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12934715#action_12934715 ] kafka0102 edited comment on SOLR-1822 at 11/22/10 9:45 PM: --- I found the problem's reason.It's tmpDir's name same to indexDir's name. tmpIdxDirName's format is SnapShooter.DATE_FMT = MMddhhmmss(hh is between 1-12). If I have a indexDirName index.20101122031000 because of a full index repl at 2010-11-22 03:10:00. And slave's repl duration is 5 minitues. So at 2010-11-22 15:10:00, the slave made indexDirName and indexDirName both index.20101122031000, and finnaly will drop the index.After that,replication would generate the above error again and again. So,just let SnapShooter.DATE_FMT = MMddHHmmss would be fine. was (Author: kafka0102): I found the problem's reason.It's tmpDir's name same to indexDir's name. tmpIdxDirName's format is SnapShooter.DATE_FMT = MMddhhmmss. If I have a indexDirName index.20101122031000 because of a full index repl at 2010-11-22 03:10:00. And slave's repl duration is 5 minitues. So at 2010-11-22 15:10:00, the slave made indexDirName and indexDirName both index.20101122031000, and finnaly will drop the index.After that,replication would generate the above error again and again. SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: Next Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Index File Format Change - LUCENE-2741
Hey folks | devs, I just wanted to give you a heads up that the just committed LUCENE-2741 contains a index file format change. Codec Files will now have a codec ID in their filenames to support codecs that use the same file extensions within a single segment. Indexes build with Lucene TRUNK will not work anymore and need to be reindexed. If you are using trunk you should reindex after you update to latest trunk. Indexes build with previous version of Lucene will work just fine and no action is required. Simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Index File Format change on trunk
Thanks for the heads-up... I tried not rebuilding the index on Solr just to see what type of exception I'd get: Caused by: java.lang.IllegalArgumentException: required codec 'tandardos Windows 7 os.version6.1sourceflushlucene.version34.0-SNAPSHOT 10' not found -Yonik http://www.lucidimagination.com On Wed, Nov 10, 2010 at 11:40 AM, Simon Willnauer simon.willna...@googlemail.com wrote: Hey folks | devs, I just wanted to give you a heads up that the just committed LUCENE-2742 contains a index file format change. Indexes build with Lucene TRUNK will not work anymore and need to be reindexed. If you are using trunk you should reindex after you update to latest trunk. Indexes build with previous version of Lucene will work just fine and no action is required. simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Index File Format change on trunk
On Thu, Nov 11, 2010 at 2:57 PM, Yonik Seeley yo...@lucidimagination.com wrote: Thanks for the heads-up... I tried not rebuilding the index on Solr just to see what type of exception I'd get: Caused by: java.lang.IllegalArgumentException: required codec 'tandardos Windows 7 os.version6.1sourceflushlucene.version34.0-SNAPSHOT 10' not found heh, it tried to read the first string in the stringstring map for the segment and get the codec for it :D yeah we didn't add a check for it since it is only a change in trunk though. simon -Yonik http://www.lucidimagination.com On Wed, Nov 10, 2010 at 11:40 AM, Simon Willnauer simon.willna...@googlemail.com wrote: Hey folks | devs, I just wanted to give you a heads up that the just committed LUCENE-2742 contains a index file format change. Indexes build with Lucene TRUNK will not work anymore and need to be reindexed. If you are using trunk you should reindex after you update to latest trunk. Indexes build with previous version of Lucene will work just fine and no action is required. simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Index File Format change on trunk
Hey folks | devs, I just wanted to give you a heads up that the just committed LUCENE-2742 contains a index file format change. Indexes build with Lucene TRUNK will not work anymore and need to be reindexed. If you are using trunk you should reindex after you update to latest trunk. Indexes build with previous version of Lucene will work just fine and no action is required. simon - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2528) CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files
[ https://issues.apache.org/jira/browse/LUCENE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-2528: Component/s: Store CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files -- Key: LUCENE-2528 URL: https://issues.apache.org/jira/browse/LUCENE-2528 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Lance Norskog Priority: Minor Attachments: LUCENE-2528.patch This patch presents a compound index file as a Lucene Directory class. This allows you to deploy one file to a query server instead of deploying a directory with the compound file and two segment files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Index file format documentation
Hi
I've read the index file format documentation (
http://lucene.apache.org/java/3_0_2/fileformats.html) trying to understand
something, and I spent quite a lot of time on the skip data documentation.
The definition that puzzled me is this:
SkipData -- SkipLevelLength, SkipLevel NumSkipLevels-1, SkipLevel
SkipDatum
First, can someone please tell me how to read this? :)
Second, I think it's incorrect (unless I misunderstood how to read it).
After reading the code, it feels to me like the definition should have been:
SkipData -- SkipLevelLength, SkipDatum NumSkipLevels-1 SkipDatum
That is, all levels 0, record their skip level data length followed by the
SkipDatum entries while level = 0 records the same besides the skip level
length ('cause it's never cached, therefore there's no need to record the
length).
Am I correct in my understanding?
Shai
Re: Index file format documentation
Actually I think the docs are [close to] correct?
SkipLevel is a sequence of SkipDatum.
Though I'm not sure why that last SkipDatum is attached; that seems odd.
Mike
On Sun, Jul 18, 2010 at 6:07 AM, Shai Erera ser...@gmail.com wrote:
Hi
I've read the index file format documentation
(http://lucene.apache.org/java/3_0_2/fileformats.html) trying to understand
something, and I spent quite a lot of time on the skip data documentation.
The definition that puzzled me is this:
SkipData -- SkipLevelLength, SkipLevel NumSkipLevels-1, SkipLevel
SkipDatum
First, can someone please tell me how to read this? :)
Second, I think it's incorrect (unless I misunderstood how to read it).
After reading the code, it feels to me like the definition should have been:
SkipData -- SkipLevelLength, SkipDatum NumSkipLevels-1 SkipDatum
That is, all levels 0, record their skip level data length followed by the
SkipDatum entries while level = 0 records the same besides the skip level
length ('cause it's never cached, therefore there's no need to record the
length).
Am I correct in my understanding?
Shai
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Index file format documentation
Ahh I've missed SkipLevel below it! Now it makes more sense.
I agree then the SkipDatum in the end is redundant … the last
SkipLevel in the definition refers to level 0.
Thanks !
Shai
On Sunday, July 18, 2010, Michael McCandless luc...@mikemccandless.com wrote:
Actually I think the docs are [close to] correct?
SkipLevel is a sequence of SkipDatum.
Though I'm not sure why that last SkipDatum is attached; that seems odd.
Mike
On Sun, Jul 18, 2010 at 6:07 AM, Shai Erera ser...@gmail.com wrote:
Hi
I've read the index file format documentation
(http://lucene.apache.org/java/3_0_2/fileformats.html) trying to understand
something, and I spent quite a lot of time on the skip data documentation.
The definition that puzzled me is this:
SkipData -- SkipLevelLength, SkipLevel NumSkipLevels-1, SkipLevel
SkipDatum
First, can someone please tell me how to read this? :)
Second, I think it's incorrect (unless I misunderstood how to read it).
After reading the code, it feels to me like the definition should have been:
SkipData -- SkipLevelLength, SkipDatum NumSkipLevels-1 SkipDatum
That is, all levels 0, record their skip level data length followed by the
SkipDatum entries while level = 0 records the same besides the skip level
length ('cause it's never cached, therefore there's no need to record the
length).
Am I correct in my understanding?
Shai
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2528) CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files
[ https://issues.apache.org/jira/browse/LUCENE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Norskog updated LUCENE-2528: -- Attachment: LUCENE-2528.patch CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files -- Key: LUCENE-2528 URL: https://issues.apache.org/jira/browse/LUCENE-2528 Project: Lucene - Java Issue Type: New Feature Reporter: Lance Norskog Priority: Minor Attachments: LUCENE-2528.patch This patch presents a compound index file as a Lucene Directory class. This allows you to deploy one file to a query server instead of deploying a directory with the compound file and two segment files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Created: (LUCENE-2528) CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files
CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files -- Key: LUCENE-2528 URL: https://issues.apache.org/jira/browse/LUCENE-2528 Project: Lucene - Java Issue Type: New Feature Reporter: Lance Norskog Priority: Minor This patch presents a compound index file as a Lucene Directory class. This allows you to deploy one file to a query server instead of deploying a directory with the compound file and two segment files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2528) CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files
[ https://issues.apache.org/jira/browse/LUCENE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12885427#action_12885427 ] Lance Norskog commented on LUCENE-2528: --- This patch has two problems: 1) CFSFileDirectory is in o.a.l.index instead of o.a.l.store because it uses SegmenInfo.commit(), which is not public. 2) It uses a hacky way to get the number of documents in the index. 3) It does not know how to get the codec, and so defaults to StandardCodec. The right way to fix #2 and #3 is to add the information to the compound file header. CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files -- Key: LUCENE-2528 URL: https://issues.apache.org/jira/browse/LUCENE-2528 Project: Lucene - Java Issue Type: New Feature Reporter: Lance Norskog Priority: Minor Attachments: LUCENE-2528.patch This patch presents a compound index file as a Lucene Directory class. This allows you to deploy one file to a query server instead of deploying a directory with the compound file and two segment files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2528) CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files
[ https://issues.apache.org/jira/browse/LUCENE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lance Norskog updated LUCENE-2528: -- Attachment: LUCENE-2528.patch CFSFileDirectory: Allow a Compound Index file to be deployed as a complete index without segment files -- Key: LUCENE-2528 URL: https://issues.apache.org/jira/browse/LUCENE-2528 Project: Lucene - Java Issue Type: New Feature Reporter: Lance Norskog Priority: Minor Attachments: LUCENE-2528.patch This patch presents a compound index file as a Lucene Directory class. This allows you to deploy one file to a query server instead of deploying a directory with the compound file and two segment files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12846099#action_12846099 ] Otis Gospodnetic commented on SOLR-1822: When Solr starts, doesn't it create the index directory? If so, this patch is not needed, unless we want to make sure replication succeeds even if someone/something removes the whole index directory on a slave after the slave had already started. Is this rally needed? SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: 1.5 Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: 1.5 SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wyhw whon updated SOLR-1822: Description: SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); was: SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: 1.5 SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wyhw whon updated SOLR-1822: Description: SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); was: SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: 1.5 SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1822) SEVERE: Unable to move index file from: tempfile to: indexfile
[ https://issues.apache.org/jira/browse/SOLR-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wyhw whon updated SOLR-1822: Attachment: SnapPuller.patch Add SnapPuller.patch SEVERE: Unable to move index file from: tempfile to: indexfile -- Key: SOLR-1822 URL: https://issues.apache.org/jira/browse/SOLR-1822 Project: Solr Issue Type: Bug Components: replication (java) Affects Versions: 1.4 Environment: Linux, JDK6,SOLR 1.4 Reporter: wyhw whon Priority: Critical Fix For: 1.5 Attachments: SnapPuller.patch SOLR index directory remvoed,but do not know what the reasons for this. I add some codes on SnapPuller.java 577 line can reslove this bug. line 576 File indexFileInIndex = new File(indexDir, fname); + if (!indexDir.exists()) indexDir.mkdir(); boolean success = indexFileInTmpDir.renameTo(indexFileInIndex); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (LUCY-66) Lucy::Util::IndexFileNames -- utilities for processing index file paths
Lucy::Util::IndexFileNames -- utilities for processing index file paths
---
Key: LUCY-66
URL: https://issues.apache.org/jira/browse/LUCY-66
Project: Lucy
Issue Type: Improvement
Components: Core
Reporter: Marvin Humphrey
Assignee: Marvin Humphrey
This implementation of IndexFileNames only supplies a handful of minor utility
functions. It has has less in it than the corresponding Lucene class, because
Lucy will delegate much more to the individual index components than Lucene.
For instance, the top level library will remain ignorant of what DocWriter
chooses to name its files. We choose never to rely on filename conventions so
that replacement components can be swapped in without requiring that they
write certain files or adhere to specific naming schemes.
The only important function provided right now is IxFileNames_extract_gen:
{code:none}
/** Skip past the first instance of an underscore in the CharBuf, then
* attempt to decode a base 36 number. For example, snapshot_5.json
* yields 5, and seg_a1 yields 27.
*
* @return a generation number, or 0 if no number can be extracted.
*/
inert i32_t
extract_gen(const CharBuf *name);
{code}
Later, after Folder is added, we'll add IxFileNames_latest_snapshot, which
will scan the root level of a Folder for the snapshot file with the highest
generation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCY-66) Lucy::Util::IndexFileNames -- utilities for processing index file paths
[
https://issues.apache.org/jira/browse/LUCY-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marvin Humphrey updated LUCY-66:
Attachment: IndexFileNames.pm
IndexFileNames.c
IndexFileNames.bp
Lucy::Util::IndexFileNames -- utilities for processing index file paths
---
Key: LUCY-66
URL: https://issues.apache.org/jira/browse/LUCY-66
Project: Lucy
Issue Type: Improvement
Components: Core
Reporter: Marvin Humphrey
Assignee: Marvin Humphrey
Attachments: IndexFileNames.bp, IndexFileNames.c, IndexFileNames.pm
This implementation of IndexFileNames only supplies a handful of minor utility
functions. It has has less in it than the corresponding Lucene class, because
Lucy will delegate much more to the individual index components than Lucene.
For instance, the top level library will remain ignorant of what DocWriter
chooses to name its files. We choose never to rely on filename conventions so
that replacement components can be swapped in without requiring that they
write certain files or adhere to specific naming schemes.
The only important function provided right now is IxFileNames_extract_gen:
{code:none}
/** Skip past the first instance of an underscore in the CharBuf, then
* attempt to decode a base 36 number. For example, snapshot_5.json
* yields 5, and seg_a1 yields 27.
*
* @return a generation number, or 0 if no number can be extracted.
*/
inert i32_t
extract_gen(const CharBuf *name);
{code}
Later, after Folder is added, we'll add IxFileNames_latest_snapshot, which
will scan the root level of a Folder for the snapshot file with the highest
generation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (LUCY-66) Lucy::Util::IndexFileNames -- utilities for processing index file paths
[
https://issues.apache.org/jira/browse/LUCY-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marvin Humphrey updated LUCY-66:
Attachment: TestIndexFileNames.c
TestIndexFileNames.bp
033-index_file_names.t
Lucy::Util::IndexFileNames -- utilities for processing index file paths
---
Key: LUCY-66
URL: https://issues.apache.org/jira/browse/LUCY-66
Project: Lucy
Issue Type: Improvement
Components: Core
Reporter: Marvin Humphrey
Assignee: Marvin Humphrey
Attachments: 033-index_file_names.t, index_file_names.diff,
IndexFileNames.bp, IndexFileNames.c, IndexFileNames.pm,
TestIndexFileNames.bp, TestIndexFileNames.c
This implementation of IndexFileNames only supplies a handful of minor utility
functions. It has has less in it than the corresponding Lucene class, because
Lucy will delegate much more to the individual index components than Lucene.
For instance, the top level library will remain ignorant of what DocWriter
chooses to name its files. We choose never to rely on filename conventions so
that replacement components can be swapped in without requiring that they
write certain files or adhere to specific naming schemes.
The only important function provided right now is IxFileNames_extract_gen:
{code:none}
/** Skip past the first instance of an underscore in the CharBuf, then
* attempt to decode a base 36 number. For example, snapshot_5.json
* yields 5, and seg_a1 yields 27.
*
* @return a generation number, or 0 if no number can be extracted.
*/
inert i32_t
extract_gen(const CharBuf *name);
{code}
Later, after Folder is added, we'll add IxFileNames_latest_snapshot, which
will scan the root level of a Folder for the snapshot file with the highest
generation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (LUCENE-877) 2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated
2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated --- Key: LUCENE-877 URL: https://issues.apache.org/jira/browse/LUCENE-877 Project: Lucene - Java Issue Type: Bug Components: Other Affects Versions: 2.1 Environment: n/a Reporter: Andreas Guther Priority: Minor I am in the process to migrate from Lucene 2.0 to Lucene 2.1. From reading the Changes document I understand that the write locks are now written into the index folder instead of the java.io.tmpdir. In the Apache Lucene - Index File Formats document in section 6.2 Lock File I read that there is a write lock used to indicate that another process is writing into the index and that this file is stored in the java.io.tempdir. This is confusing to me. I had the impression all lock files go into the index folder now. And using the the java.io.tempdir is only local and does not support access to shared index folders. Do I miss something here or is the documentation not updated? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Assigned: (LUCENE-877) 2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated
[ https://issues.apache.org/jira/browse/LUCENE-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless reassigned LUCENE-877: - Assignee: Michael McCandless 2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated --- Key: LUCENE-877 URL: https://issues.apache.org/jira/browse/LUCENE-877 Project: Lucene - Java Issue Type: Bug Components: Other Affects Versions: 2.1 Environment: n/a Reporter: Andreas Guther Assigned To: Michael McCandless Priority: Minor I am in the process to migrate from Lucene 2.0 to Lucene 2.1. From reading the Changes document I understand that the write locks are now written into the index folder instead of the java.io.tmpdir. In the Apache Lucene - Index File Formats document in section 6.2 Lock File I read that there is a write lock used to indicate that another process is writing into the index and that this file is stored in the java.io.tempdir. This is confusing to me. I had the impression all lock files go into the index folder now. And using the the java.io.tempdir is only local and does not support access to shared index folders. Do I miss something here or is the documentation not updated? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Resolved: (LUCENE-877) 2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated
[ https://issues.apache.org/jira/browse/LUCENE-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless resolved LUCENE-877. --- Resolution: Fixed Fix Version/s: 2.2 2.1 Locking documentation in Apache Lucene - Index File Formats section 6.2 Lock File out dated --- Key: LUCENE-877 URL: https://issues.apache.org/jira/browse/LUCENE-877 Project: Lucene - Java Issue Type: Bug Components: Other Affects Versions: 2.1 Environment: n/a Reporter: Andreas Guther Assigned To: Michael McCandless Priority: Minor Fix For: 2.2 I am in the process to migrate from Lucene 2.0 to Lucene 2.1. From reading the Changes document I understand that the write locks are now written into the index folder instead of the java.io.tmpdir. In the Apache Lucene - Index File Formats document in section 6.2 Lock File I read that there is a write lock used to indicate that another process is writing into the index and that this file is stored in the java.io.tempdir. This is confusing to me. I had the impression all lock files go into the index folder now. And using the the java.io.tempdir is only local and does not support access to shared index folders. Do I miss something here or is the documentation not updated? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Cutting updated HADOOP-984: Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks, Arun! Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 Attachments: HADOOP-984_20070207_1.patch The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Priority: Critical The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HADOOP-984: - Fix Version/s: 0.11.1 Assignee: Arun C Murthy Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HADOOP-984: - Attachment: HADOOP-984_20070207_1.patch Simple fix - moved the offending code into the appropriate try-block. Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 Attachments: HADOOP-984_20070207_1.patch The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated HADOOP-984: - Status: Patch Available (was: Open) Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 Attachments: HADOOP-984_20070207_1.patch The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470816 ] Hadoop QA commented on HADOOP-984: -- +1, because http://issues.apache.org/jira/secure/attachment/12350518/HADOOP-984_20070207_1.patch applied and successfully tested against trunk revision r504381. Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 Attachments: HADOOP-984_20070207_1.patch The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-984) Exception while retrieving a map output index file is not reported to JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470825 ] Owen O'Malley commented on HADOOP-984: -- +1 Exception while retrieving a map output index file is not reported to JobTracker Key: HADOOP-984 URL: https://issues.apache.org/jira/browse/HADOOP-984 Project: Hadoop Issue Type: Bug Components: mapred Affects Versions: 0.11.0 Reporter: Nigel Daley Assigned To: Arun C Murthy Priority: Critical Fix For: 0.11.1 Attachments: HADOOP-984_20070207_1.patch The index file of a map output reported a checksum error and was moved to bad_file dir. Reduces, however, continued to attempt retrieval of the file forever. It seem the checksum exception did *NOT* result in the map's TaskTracker reporting to the JobTracker that the map output was lost. This was likely introduced by HADOOP-331. TaskTracker log: 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: Moving bad file /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to /foo/bad_files/file.out.index.-48633003 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) at org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) at org.mortbay.http.HttpServer.service(HttpServer.java:954) at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
[ http://issues.apache.org/jira/browse/LUCENE-706?page=all ] Grant Ingersoll resolved LUCENE-706. Resolution: Fixed Applied. Thanks Johan and Doron. Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Grant Ingersoll Priority: Trivial Attachments: file-format-frq-example.patch Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Assigned: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
[ http://issues.apache.org/jira/browse/LUCENE-706?page=all ] Grant Ingersoll reassigned LUCENE-706: -- Assignee: Grant Ingersoll (was: Doron Cohen) Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Grant Ingersoll Priority: Trivial Attachments: file-format-frq-example.patch Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
[ http://issues.apache.org/jira/browse/LUCENE-706?page=comments#action_12447044 ] Steven Rowe commented on LUCENE-706: Hex: 0D is NOT the same as decimal 15. 0Dh = 13d. 15d = 0Fh. Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Grant Ingersoll Priority: Trivial Attachments: file-format-frq-example.patch Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
[ http://issues.apache.org/jira/browse/LUCENE-706?page=comments#action_12447049 ] Doron Cohen commented on LUCENE-706: Right, sorry, copied that hex data from an .frq of an index with a different example, where the frequencies were 1 in doc 6 and 3 in doc 10, so there you would get 2 * 6 + 1 = 13. For the correct example of freq 1 in doc 7 and 3 in doc 11 the .frq content is 0F 08 03 as it should be. (Meaning that the documentatin should still be fixed...;-) Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Grant Ingersoll Priority: Trivial Attachments: file-format-frq-example.patch Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Created: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Doron Cohen Priority: Trivial Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-706) Index File Format - Example for frequency file .frq is wrong
[ http://issues.apache.org/jira/browse/LUCENE-706?page=all ] Doron Cohen updated LUCENE-706: --- Lucene Fields: [New, Patch Available] (was: [New]) Index File Format - Example for frequency file .frq is wrong Key: LUCENE-706 URL: http://issues.apache.org/jira/browse/LUCENE-706 Project: Lucene - Java Issue Type: Improvement Components: Website Environment: not applicable Reporter: Doron Cohen Assigned To: Doron Cohen Priority: Trivial Attachments: file-format-frq-example.patch Reported by Johan Stuyts - http://www.nabble.com/Possible-documentation-error--p7012445.html - Frequency file example says: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 22, 3 It should be: For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven would be the following sequence of VInts: 15, 8, 3 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
