[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[ https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-7168: - Fix Version/s: (was: 6.0) master (7.0) Manually correcting fixVersion per Step #S5 of LUCENE-7271 > Remove geo3d test leniency > -- > > Key: LUCENE-7168 > URL: https://issues.apache.org/jira/browse/LUCENE-7168 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Fix For: 6.1, master (7.0) > > Attachments: LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch, > LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch > > > Today the test hides possible failures by leniently handling quantization > issues. > We should fix it to do what geo2d tests now do: pre-quantized indexed points, > but don't quantize query shapes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
Here's a new patch, simplifying the encode/decode, based on a good
idea that [~daddywri] suggested (over lunch!) to change decode to decode
to the center value. This gets us our stability without having to
pick special double values, and it means we can use the full integer
space again, but I did need to add the same if that {{LatLonPoint}}
has about not being able to encode precisely the max value.
> Remove geo3d test leniency
> --
>
> Key: LUCENE-7168
> URL: https://issues.apache.org/jira/browse/LUCENE-7168
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: master, 6.1
>
> Attachments: LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch,
> LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch
>
>
> Today the test hides possible failures by leniently handling quantization
> issues.
> We should fix it to do what geo2d tests now do: pre-quantized indexed points,
> but don't quantize query shapes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
OK how about this? I moved the min/max decode back to {{Geo3DUtil}} (package
private!), renamed them to floor/ceil.
I'm beasting now...
Sorry for the noise [~daddywri]! Now we can have a relaxed lunch :)
> Remove geo3d test leniency
> --
>
> Key: LUCENE-7168
> URL: https://issues.apache.org/jira/browse/LUCENE-7168
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Fix For: master, 6.1
>
> Attachments: LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch,
> LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch
>
>
> Today the test hides possible failures by leniently handling quantization
> issues.
> We should fix it to do what geo2d tests now do: pre-quantized indexed points,
> but don't quantize query shapes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
OK, another patch, addressing all nocommits. I think it's nearly
ready... I had to an "if" to decode to never go below {{-planetMax}}.
However the above failure still happens ...
So I've improved this test so that on failure, it runs a crazy (test
only) {{IntersectVisitor}} that "explains" what happened to this one
hit. For this failure, it now produces output like this:
{noformat}
[junit4] Started J0 PID(11250@localhost).
[junit4] Suite: org.apache.lucene.spatial3d.TestGeo3DPoint
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeo3DPoint
-Dtests.method=testRandomBig -Dtests.seed=3AD1F04EDC75F5CC -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=lucenedata/hudson.enwiki.random.lines.txt.fixed
-Dtests.locale=ru-RU -Dtests.timezone=America/Grand_Turk -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 5.47s | TestGeo3DPoint.testRandomBig <<<
[junit4]> Throwable #1: java.lang.AssertionError: FAIL: id=541300 should
have matched but did not
[junit4]> shape=GeoStandardCircle: {planetmodel=PlanetModel.WGS84,
center=[lat=-0.8971654677124566, lon=-0.3398482030102755],
radius=1.4775317506492547(84.65633340877824)}
[junit4]> point=[X=0.8653002868649471, Y=0.50134342478497,
Z=0.046203414829601996]
[junit4]> docID=538760 deleted?=false
[junit4]> query=PointInGeo3DShapeQuery: field=point: Shape:
GeoStandardCircle: {planetmodel=PlanetModel.WGS84,
center=[lat=-0.8971654677124566, lon=-0.3398482030102755],
radius=1.4775317506492547(84.65633340877824)}
[junit4]> explanation:
[junit4]> target is in leaf _2u(7.0.0):c1 of full reader
StandardDirectoryReader(segments:210:nrt _12(7.0.0):c198337/1995:delGen=1
_24(7.0.0):c199685/601:delGen=1 _23(7.0.0):c5413/17:delGen=2
_25(7.0.0):c5413/12:delGen=1 _26(7.0.0):c5413/10:delGen=1
_27(7.0.0):c5413/10:delGen=1 _28(7.0.0):c5413/17:delGen=2
_29(7.0.0):c5413/8:delGen=2 _2a(7.0.0):c5413/12:delGen=1
_2b(7.0.0):c5413/14:delGen=1 _2c(7.0.0):c5413/4:delGen=1
_2d(7.0.0):c5413/13:delGen=2 _2e(7.0.0):c5413/12:delGen=2
_2f(7.0.0):c5413/15:delGen=2 _2g(7.0.0):c5413/10:delGen=2
_2h(7.0.0):c5413/9:delGen=2 _2i(7.0.0):c5413/6:delGen=1
_2j(7.0.0):c5413/9:delGen=1 _2k(7.0.0):c5413/8:delGen=1
_2l(7.0.0):c5413/5:delGen=1 _2m(7.0.0):c5413/3:delGen=1
_2n(7.0.0):c5413/4:delGen=1 _2o(7.0.0):c5413 _2p(7.0.0):c5413/5:delGen=1
_2q(7.0.0):c5413/1:delGen=1 _2r(7.0.0):c5413 _2s(7.0.0):c5413 _2t(7.0.0):c5413
_2u(7.0.0):c1)
[junit4]> full BKD path to target doc:
[junit4]> Cell(x=0.8653002866318559 TO 0.8653002870980383
y=0.5013434245518787 TO 0.5013434250180612 z=0.04620341459651078 TO
0.04620341506269321)
[junit4]> on cell Cell(x=0.8653002866318559 TO 0.8653002870980383
y=0.5013434245518787 TO 0.5013434250180612 z=0.04620341459651078 TO
0.04620341506269321), wrapped visitor returned CELL_OUTSIDE_QUERY
[junit4]>at
__randomizedtesting.SeedInfo.seed([3AD1F04EDC75F5CC:BD868DC14D2C894C]:0)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.verify(TestGeo3DPoint.java:741)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.doTestRandom(TestGeo3DPoint.java:537)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomBig(TestGeo3DPoint.java:469)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at:
/l/geo3dquant/lucene/build/spatial3d/test/J0/temp/lucene.spatial3d.TestGeo3DPoint_3AD1F04EDC75F5CC-026
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60):
{id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))},
docValues:{id=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=357,
maxMBSortInHeap=5.860921451607786, sim=ClassicSimilarity, locale=ru-RU,
timezone=America/Grand_Turk
[junit4] 2> NOTE: Linux 3.13.0-71-generic amd64/Oracle Corporation
1.8.0_60 (64-bit)/cpus=8,threads=1,free=410208728,total=495452160
[junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPoint]
[junit4] Completed [1/1 (1!)] in 5.63s, 1 test, 1 failure <<< FAILURES!
{noformat}
See the new "explanation" part. Anyway, I was then able to turn those
BKD traversal details into a standalone test, that does fail:
{noformat}
public void testCuriousFailure() throws Exception {
GeoShape shape = GeoCircleFactory.makeGeoCircle(PlanetModel.WGS84,
-0.8971654677124566, -0.3398482030102755, 1.4775317506492547);
GeoPoint point = new GeoPoint(0.8653002868649471, 0.50134342478497,
0.046203414829601996);
// point is inside our circle shape:
assertTrue(shape.isWithin(point));
double xMin = 0.8653002866318559;
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
OK, some progress (new patch).
I created a silly method to find a "good" double value for "quantizing to int"
purposes:
{noformat}
/** Returns a double value >= x such that if you multiply that value by an
int, and then
* divide it by that int again, you get precisely the same value back */
{noformat}
If we use such a value, then the encode/decode is stable. E.g. for WGS84, the
{{planetMax}} is 1.0011188539924791, and the nextSafeDouble after that value is
1.0011191368103027.
But there is still one problem that I'm trying to work out, which is that if
you encode the min value ({{-planetMax}}), and then decode that, you get back a
value LESS than {{-planetMax}}, because of the flooring we do, and it's not
guaranteed the planetMax would "be" at a floor boundary. Somehow, magically,
{{LatLonPoint}}'s min values seem to work ;)
Other tests are passing, except with this patch I hit this failure on beasting:
{noformat}
[junit4] Started J0 PID(5288@localhost).
[junit4] Suite: org.apache.lucene.spatial3d.TestGeo3DPoint
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeo3DPoint
-Dtests.method=testRandomBig -Dtests.seed=3AD1F04EDC75F5CC -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed
-Dtests.locale=ru-RU -Dtests.timezone=America/Grand_Turk -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 5.05s | TestGeo3DPoint.testRandomBig <<<
[junit4]> Throwable #1: java.lang.AssertionError: FAIL: id=541300 should
have matched but did not
[junit4]> shape=GeoStandardPath: {planetmodel=PlanetModel.WGS84,
width=0.3665191429188092(21.0), points={[[lat=-0.9764909432864418,
lon=1.5923866750547349], [lat=-0.044905310521497856, lon=0.543041341355748],
[lat=-1.50783500438853, lon=-0.8877147957518042]]}}
[junit4]> point=[X=0.8653002868649471, Y=0.50134342478497,
Z=0.046203414829601996]
[junit4]> docID=539356 deleted?=false
[junit4]> query=PointInGeo3DShapeQuery: field=point: Shape:
GeoStandardPath: {planetmodel=PlanetModel.WGS84,
width=0.3665191429188092(21.0), points={[[lat=-0.9764909432864418,
lon=1.5923866750547349], [lat=-0.044905310521497856, lon=0.543041341355748],
[lat=-1.50783500438853, lon=-0.8877147957518042]]}}
[junit4]>at
__randomizedtesting.SeedInfo.seed([3AD1F04EDC75F5CC:BD868DC14D2C894C]:0)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.verify(TestGeo3DPoint.java:730)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.doTestRandom(TestGeo3DPoint.java:530)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomBig(TestGeo3DPoint.java:462)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at:
/l/geo3dquant/lucene/build/spatial3d/test/J0/temp/lucene.spatial3d.TestGeo3DPoint_3AD1F04EDC75F5CC-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60):
{id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))},
docValues:{id=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=357,
maxMBSortInHeap=5.860921451607786, sim=ClassicSimilarity, locale=ru-RU,
timezone=America/Grand_Turk
[junit4] 2> NOTE: Linux 3.13.0-71-generic amd64/Oracle Corporation
1.8.0_60 (64-bit)/cpus=8,threads=1,free=407941736,total=492306432
[junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPoint]
[junit4] Completed [1/1 (1!)] in 5.23s, 1 test, 1 failure <<< FAILURES!
{noformat}
I haven't dug into it yet ...
> Remove geo3d test leniency
> --
>
> Key: LUCENE-7168
> URL: https://issues.apache.org/jira/browse/LUCENE-7168
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Attachments: LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch,
> LUCENE-7168.patch
>
>
> Today the test hides possible failures by leniently handling quantization
> issues.
> We should fix it to do what geo2d tests now do: pre-quantized indexed points,
> but don't quantize query shapes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
Work-in-progress, dirty patch, fixing (I think) the test quantization.
I also attempted to fix geo3d's encode/decode to look like {{LatLonPoint}}, and
added some encode/decode tests, however
{{TestGeo3DPoint.testEncodeDecodeIsStable}} is failing!
I also moved some "called only by tests" methods into the test cases, and
"called only by the one query" methods into the query.
> Remove geo3d test leniency
> --
>
> Key: LUCENE-7168
> URL: https://issues.apache.org/jira/browse/LUCENE-7168
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Attachments: LUCENE-7168.patch, LUCENE-7168.patch, LUCENE-7168.patch
>
>
> Today the test hides possible failures by leniently handling quantization
> issues.
> We should fix it to do what geo2d tests now do: pre-quantized indexed points,
> but don't quantize query shapes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[
https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-7168:
---
Attachment: LUCENE-7168.patch
Another patch, just improving the failure message, and beasting uncovered a
failure!
{noformat}
[junit4] Suite: org.apache.lucene.spatial3d.TestGeo3DPoint
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeo3DPoint
-Dtests.method=testRandomMedium -Dtests.seed=1B44C9C6ED816CF9 -Dtests.slow=true
-Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed
-Dtests.locale=pl-PL -Dtests.timezone=Asia/Damascus -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.68s | TestGeo3DPoint.testRandomMedium <<<
[junit4]> Throwable #1: java.lang.AssertionError: FAIL: id=147 should
have matched but did not
[junit4]> shape=GeoPath: {planetmodel=PlanetModel.WGS84,
width=0.9250245035569946(53.0), points={[[lat=-0.31202581053256323,
lon=-2.3769151553645593], [lat=-0.04559091474230697, lon=0.3560334442429531],
[lat=-0.6470509953978505, lon=1.0469362861181948], [lat=-0.685120355322366,
lon=1.6043555643894918], [lat=-1.568917110871099, lon=1.008031992795193]]}}
[junit4]> lat=2.3507843431772453 lon=127.59305012508766
[junit4]> point=[lat=0.04102892679277523, lon=2.2269188273449423]
[junit4]> docID=133 deleted?=false
[junit4]> query=PointInGeo3DShapeQuery: field=point: Shape: GeoPath:
{planetmodel=PlanetModel.WGS84, width=0.9250245035569946(53.0),
points={[[lat=-0.31202581053256323, lon=-2.3769151553645593],
[lat=-0.04559091474230697, lon=0.3560334442429531], [lat=-0.6470509953978505,
lon=1.0469362861181948], [lat=-0.685120355322366, lon=1.6043555643894918],
[lat=-1.568917110871099, lon=1.008031992795193]]}}
[junit4]>at
__randomizedtesting.SeedInfo.seed([1B44C9C6ED816CF9:A69AFE6EACE40F9F]:0)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.verify(TestGeo3DPoint.java:741)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.doTestRandom(TestGeo3DPoint.java:529)
[junit4]>at
org.apache.lucene.spatial3d.TestGeo3DPoint.testRandomMedium(TestGeo3DPoint.java:456)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene60):
{id=PostingsFormat(name=Asserting)},
docValues:{id=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=1169,
maxMBSortInHeap=6.9707915551209005, sim=ClassicSimilarity, locale=pl-PL,
timezone=Asia/Damascus
[junit4] 2> NOTE: Linux 3.13.0-71-generic amd64/Oracle Corporation
1.8.0_60 (64-bit)/cpus=8,threads=1,free=456488064,total=504889344
[junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPoint]
[junit4] Completed [1/1 (1!)] in 0.84s, 1 test, 1 failure <<< FAILURES!
{noformat}
> Remove geo3d test leniency
> --
>
> Key: LUCENE-7168
> URL: https://issues.apache.org/jira/browse/LUCENE-7168
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Attachments: LUCENE-7168.patch, LUCENE-7168.patch
>
>
> Today the test hides possible failures by leniently handling quantization
> issues.
> We should fix it to do what geo2d tests now do: pre-quantized indexed points,
> but don't quantize query shapes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7168) Remove geo3d test leniency
[ https://issues.apache.org/jira/browse/LUCENE-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-7168: --- Attachment: LUCENE-7168.patch Here's a patch to remove query-time quantizing, threads, and the "small" boolean. Tests seem to survive beasting so far ... > Remove geo3d test leniency > -- > > Key: LUCENE-7168 > URL: https://issues.apache.org/jira/browse/LUCENE-7168 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-7168.patch > > > Today the test hides possible failures by leniently handling quantization > issues. > We should fix it to do what geo2d tests now do: pre-quantized indexed points, > but don't quantize query shapes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
