[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Updated patch: * Quantizes random generated lat/lon values to hash scale * Reverts overly complicated compare method * Adds @Nightly to repeated adversarial tests * Passes all beast tests * Passes ant precommit I think this is ready and we can move on to fixing performance by removing some of these transient structures (e.g., rangeList) > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Patch of lucene6780 feature branch against trunk. I think its ready to merge. > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Updated patch to remove BKDDistanceQuery and make randomLat/Lon non-static > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch git svn is behind by 3 days... here's the patch against the latest svn repo. > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Updated patch. * Adds Nightly to a few of the tests that still seem to run slow during range creation (particularly on PointInPolygon queries). IMHO this shouldn't preclude the fixes and improvements added by this patch. Will open separate issues to a. investigate performance of range/polygon intersection (probably needs to be handled under the dateline crossing feature) and b. create ranges on-demand instead of using transient memory (improve overall performance) * Added consistency to rectContainsPoint to use same method as GeoUtils.bboxContains > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Next WIP patch... This patch includes the following: 1. Throws an IllegalArgumentException for PointDistanceQuery when the radius exceeds the maximum allowable distance (based on location on the earth) 2. Reduces resolution for extreme GeoPointInBBoxQuery cases (based on diagonal distance of the bbox). Performance is much better (need to quantify through benchmarks) but there are still a few Test bugs that need to be ironed out (PointInPolygon tolerance, and catching illegal radius values for PointDistanceQuery) > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[
https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-6780:
---
Attachment: LUCENE-6780-heap-used-hack.patch
I was concerned about the peak heap used for "worst case" queries here, so I
hacked up a quick patch (attached) to test this:
{noformat}
GeoPointInBBoxQuery: field=point: Lower Left:
[-170.79577068315723,-88.3524701239041] Upper Right:
[115.75692731020496,51.78004322487766]
--> 940,015 terms = 34,065,696 bytes
GeoPointDistanceQuery: field=point: Center:
[-95.87683480747508,-83.99672364681616] Distance: 826889.911703281 meters]
--> 179,562 terms = 12,446,344 bytes
{noformat}
The patch just records over time the largest number of terms created by the
query, and then I ran the test for many iterations.
I think this is too high, e.g. too many of these queries in flight at once can
mean an unexpected OOME.
But I think before we address this we should address the correctness issues
(the failing seeds for {{TestGeoUtils.testGeoRelations}}).
It could be that to fix these, we place soft limits on how large each query is
allowed to be? Meaning, a user who's willing to have more error, willing to
use more heap, can increase the limit if they want, but by default the limit
protects the more common use case with smaller shapes.
> GeoPointDistanceQuery doesn't work with a large radius?
> ---
>
> Key: LUCENE-6780
> URL: https://issues.apache.org/jira/browse/LUCENE-6780
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Attachments: LUCENE-6780-heap-used-hack.patch, LUCENE-6780.patch,
> LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch
>
>
> I'm working on LUCENE-6698 but struggling with test failures ...
> Then I noticed that TestGeoPointQuery's test never tests on large distances,
> so I modified the test to sometimes do so (like TestBKDTree) and hit test
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Next iteration patch off the feature branch. bq. OK I committed a new randomized test The test was buggy using maxLon where it expected minLon. I also updated it to handle the case where the circle is fully contained by the rectangle. All beasting passed. bq. I'm seeing very slow query execution times I had noticed this before posting the initial patch and isolated one test that took ~45 secs to complete. The slow times were related to recomputing the bbox and high resolution ranges for every segment on very large distance queries. The new patch relaxes the distance criteria for the range resolution. There are still some boundary outliers that take ~15 seconds (instead of 60+) to complete. I can further improve this by optimizing the GeoPointDistanceQuery.computeBBox method... or, what are your thoughts on computing the BBox and reusing across segments?? bq. I still think there's really a bug here I committed a failing test on the branch bleh...indeed!! Fixed it up. > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch There's a small change to GeoPointTermsEnum in LUCENE-6777 that this depends upon. Apply LUCENE-6777.patch then LUCENE-6780.patch This patch also fixes an issue noted in LUCENE-6698 (a side effect of simplifying the cellCrossesCircle logic) > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Knize updated LUCENE-6780: --- Attachment: LUCENE-6780.patch New patch that computes the BBox given the WGS84 ellipsoid. This skips all haversine and vincenty issues at the expense of a few extra trig functions (only used to compute bbox once). The patch also addresses other comments (e.g., exploding ternary operator, adding java doc). nocommits are left alone for time being. > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch, > LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[ https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-6780: --- Attachment: LUCENE-6780.patch Patch combining [~nknize]'s last patch and the original "make test more evil" patch. > GeoPointDistanceQuery doesn't work with a large radius? > --- > > Key: LUCENE-6780 > URL: https://issues.apache.org/jira/browse/LUCENE-6780 > Project: Lucene - Core > Issue Type: Bug >Reporter: Michael McCandless > Attachments: LUCENE-6780.patch, LUCENE-6780.patch, LUCENE-6780.patch > > > I'm working on LUCENE-6698 but struggling with test failures ... > Then I noticed that TestGeoPointQuery's test never tests on large distances, > so I modified the test to sometimes do so (like TestBKDTree) and hit test > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6780) GeoPointDistanceQuery doesn't work with a large radius?
[
https://issues.apache.org/jira/browse/LUCENE-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-6780:
---
Attachment: LUCENE-6780.patch
Patch, putting back the {{smallBBox}} boolean ...
First problem is the test runs very slowly sometimes. I thought LUCENE-6712
was supposed to fix this (so we don't need to do LUCENE-6685)?
Second problem is it then hits failures:
{noformat}
[junit4:pickseed] Seed property 'tests.seed' already defined: EE938F778B784339
[junit4] says hallo! Master seed: EE938F778B784339
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(7661@localhost).
[junit4] Suite: org.apache.lucene.search.TestGeoPointQuery
[junit4] 1> T0: id=9 docID=9 lat=-76.34720742943381 lon=-36.15344492760056
deleted?=false expected=true but got false query=GeoPointDistanceQuery:
field=geoField: Center: [-91.32459786462451,-62.68808104026591] Distance:
2652065.508673892 m Lower Left: [-135.1399256521194,-86.45202902120825] Upper
Right: [-47.50927007712964,-38.847306191642886]
[junit4] 2> 9ζ 03, 2015 11:39:46 δΈε
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
uncaughtException
[junit4] 2> WARNING: Uncaught exception in thread:
Thread[T0,5,TGRP-TestGeoPointQuery]
[junit4] 2> java.lang.AssertionError: wrong hit
[junit4] 2>at
__randomizedtesting.SeedInfo.seed([EE938F778B784339]:0)
[junit4] 2>at org.junit.Assert.fail(Assert.java:93)
[junit4] 2>at
org.apache.lucene.search.TestGeoPointQuery$VerifyHits.test(TestGeoPointQuery.java:570)
[junit4] 2>at
org.apache.lucene.search.TestGeoPointQuery$1._run(TestGeoPointQuery.java:511)
[junit4] 2>at
org.apache.lucene.search.TestGeoPointQuery$1.run(TestGeoPointQuery.java:402)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeoPointQuery
-Dtests.method=testRandomTiny -Dtests.seed=EE938F778B784339 -Dtests.slow=true
-Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed
-Dtests.locale=zh_HK -Dtests.timezone=Portugal -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 63.4s | TestGeoPointQuery.testRandomTiny <<<
{noformat}
Is it known/expected that the math doesn't work for large distance queries?
Finally, I noticed the randomized test is not testing "crosses dateline" cases
.. is this supposed to work / known not to work? I can open a separate issue
for that.
> GeoPointDistanceQuery doesn't work with a large radius?
> ---
>
> Key: LUCENE-6780
> URL: https://issues.apache.org/jira/browse/LUCENE-6780
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
> Attachments: LUCENE-6780.patch
>
>
> I'm working on LUCENE-6698 but struggling with test failures ...
> Then I noticed that TestGeoPointQuery's test never tests on large distances,
> so I modified the test to sometimes do so (like TestBKDTree) and hit test
> failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
