[jira] [Comment Edited] (LUCENE-6829) OfflineSorter should use Directory API
[
https://issues.apache.org/jira/browse/LUCENE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962819#comment-14962819
]
Steve Rowe edited comment on LUCENE-6829 at 10/19/15 4:24 AM:
--
Another test failure my Jenkins found with the same root cause, only reproduces
8/100 iters for me:
{noformat}
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterCommit -Dtests.method=testPrepareCommitRollback
-Dtests.seed=98FCDFF6002C97E1 -Dtests.nightly=true -Dtests.slow=true
-Dtests.linedocsfile=/home/jenkins/lucene-data/e
nwiki.random.lines.txt -Dtests.locale=is -Dtests.timezone=Australia/Sydney
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.32s | TestIndexWriterCommit.testPrepareCommitRollback <<<
[junit4]> Throwable #1: java.lang.NullPointerException
[junit4]>at
__randomizedtesting.SeedInfo.seed([98FCDFF6002C97E1:94F93159522F894E]:0)
[junit4]>at
java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
[junit4]>at
java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
[junit4]>at java.util.Arrays.sort(Arrays.java:1246)
[junit4]>at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:660)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:73)
[junit4]>at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
[junit4]>at
org.apache.lucene.index.TestIndexWriterCommit.testPrepareCommitRollback(TestIndexWriterCommit.java:599)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene53):
{content=Lucene50(blocksize=128)}, docValues:{},
sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {content=DFR
I(ne)3(800.0)}, locale=is, timezone=Australia/Sydney
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
1.8.0_45 (64-bit)/cpus=16,threads=1,free=401791056,total=514850816
[junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterCommit]
{noformat}
was (Author: steve_rowe):
Another test failure my Jenkins found with the same root cause, only reproduces
8/100 iters for me:
{noformat}
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterCommit -Dtests.method=testPrepareCommitRollback
-Dtests.seed=98FCDFF6002C97E1 -Dtests.nightly=true -Dtests.slow=true
-Dtests.linedocsfile=/home/jenkins/lucene-data/e
nwiki.random.lines.txt -Dtests.locale=is -Dtests.timezone=Australia/Sydney
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.32s | TestIndexWriterCommit.testPrepareCommitRollback <<<
[junit4]> Throwable #1: java.lang.NullPointerException
[junit4]>at __randomizedtesting.SeedInfo.seed([98FCDFF6002C97E1:9
4F93159522F894E]:0)
[junit4]>at java.util.ComparableTimSort.binarySort(ComparableTimS
ort.java:258)
[junit4]>at java.util.ComparableTimSort.sort(ComparableTimSort.ja
va:203)
[junit4]>at java.util.Arrays.sort(Arrays.java:1246)
[junit4]>at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:660)
[junit4]>at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:73)
[junit4]>at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
[junit4]>at
org.apache.lucene.index.TestIndexWriterCommit.testPrepareCommitRollback(TestIndexWriterCommit.java:599)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene53):
{content=Lucene50(blocksize=128)}, docValues:{},
sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {content=DFR
I(ne)3(800.0)}, locale=is, timezone=Australia/Sydney
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
1.8.0_45 (64-bit)/cpus=16,threads=1,free=401791056,total=514850816
[junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterCommit]
{noformat}
> OfflineSorter should use Directory API
> --
>
> Key: LUCENE-6829
> URL: https://issues.apache.org/jira/browse/LUCENE-6829
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk
>
> Attachments: LUCENE-6829.patch, LUCENE-6829.patch, LUCENE-6829.patch,
> LUCENE-6829.patch
>
>
> I think this is a blocker for LUCENE-6825, because the block KD-tree makes
> heavy use of OfflineSorter and we don't want to fill up tmp space ...
> This should be a straightforward cutover, but there are some challenges, e.g.
> the test was failing because virus
[jira] [Comment Edited] (LUCENE-6829) OfflineSorter should use Directory API
[
https://issues.apache.org/jira/browse/LUCENE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954520#comment-14954520
]
Dawid Weiss edited comment on LUCENE-6829 at 10/13/15 7:04 AM:
---
This is a large patch and I only scanned it briefly. It looks good to me. I
don't know how to avoid the virus checker special case (requiring odd hacks in
the code to disable it).
Also, blocks like this one:
{code}
+ Path tempPath =
Files.createTempDirectory(Dictionary.getDefaultTempDir(), "Hunspell");
+ boolean success = false;
+ try (Directory tempDir = FSDirectory.open(tempPath)) {
+this.dictionary = new Dictionary(tempDir, "hunspell", affix,
dictionaries, ignoreCase);
+success = true;
+ } finally {
+// tempPath (directory) should be empty at this point:
+if (success) {
+ Files.delete(tempPath);
+} else {
+ IOUtils.deleteFilesIgnoringExceptions(tempPath);
+}
+ }
{code}
Is there any reason why we shouldn't just let the regular exception suppression
be used here? I know it'd reverse the precedence, but at least you'd get the
full picture (temp. file couldn't be deleted too). Isn't this a leftover
pattern from before 1.7 days?
So, to be clear, why isn't the above just:
{code}
+ Path tempPath =
Files.createTempDirectory(Dictionary.getDefaultTempDir(), "Hunspell");
+ try (Directory tempDir = FSDirectory.open(tempPath)) {
+this.dictionary = new Dictionary(tempDir, "hunspell", affix,
dictionaries, ignoreCase);
+ } finally {
+ Files.delete(tempPath); // will fail if tempPath has any files in it,
suppressing any exception.
+ }
{code}
was (Author: dweiss):
This is a large patch and I only scanned it briefly. It looks good to me. I
don't know how to avoid the virus checker special case (requiring odd hacks in
the code to disable it).
Also, blocks like this one:
{code}
+ Path tempPath =
Files.createTempDirectory(Dictionary.getDefaultTempDir(), "Hunspell");
+ boolean success = false;
+ try (Directory tempDir = FSDirectory.open(tempPath)) {
+this.dictionary = new Dictionary(tempDir, "hunspell", affix,
dictionaries, ignoreCase);
+success = true;
+ } finally {
+// tempPath (directory) should be empty at this point:
+if (success) {
+ Files.delete(tempPath);
+} else {
+ IOUtils.deleteFilesIgnoringExceptions(tempPath);
+}
+ }
{code}
Is there any reason why we shouldn't just let the regular exception suppression
be used here? I know it'd reverse the precedence, but at least you'd get the
full picture (temp. file couldn't be deleted too). Isn't this a leftover
pattern from before 1.7 days?
> OfflineSorter should use Directory API
> --
>
> Key: LUCENE-6829
> URL: https://issues.apache.org/jira/browse/LUCENE-6829
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6829.patch, LUCENE-6829.patch, LUCENE-6829.patch,
> LUCENE-6829.patch
>
>
> I think this is a blocker for LUCENE-6825, because the block KD-tree makes
> heavy use of OfflineSorter and we don't want to fill up tmp space ...
> This should be a straightforward cutover, but there are some challenges, e.g.
> the test was failing because virus checker blocked deleting of files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
