[jira] [Commented] (LUCENE-7355) Leverage MultiTermAwareComponent in query parsers
[ https://issues.apache.org/jira/browse/LUCENE-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722299#comment-16722299 ] ASF GitHub Bot commented on LUCENE-7355: GitHub user xristy opened a pull request: https://github.com/apache/jena/pull/512 merge analyzer-regression per LUCENE-7355 add normalize to LowerCaseKeywordAnalyzer You can merge this pull request into a Git repository by running: $ git pull https://github.com/buda-base/jena JENA-1652-analyzer-regression Alternatively you can review and apply these changes as the patch at: https://github.com/apache/jena/pull/512.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #512 commit e63b85fdc0514de7916998ef15e4760f2449574f Author: Code Ferret Date: 2018-12-15T21:41:38Z merge analyzer-regression > Leverage MultiTermAwareComponent in query parsers > - > > Key: LUCENE-7355 > URL: https://issues.apache.org/jira/browse/LUCENE-7355 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Assignee: Adrien Grand >Priority: Minor > Fix For: 6.2, 7.0 > > Attachments: LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, > LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, > LUCENE-7355.patch > > > MultiTermAwareComponent is designed to make it possible to do the right thing > in query parsers when in comes to analysis of multi-term queries. However, > since query parsers just take an analyzer and since analyzers do not > propagate the information about what to do for multi-term analysis, query > parsers cannot do the right thing out of the box. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11617) Expose Alias Metadata CRUD in REST API
[ https://issues.apache.org/jira/browse/SOLR-11617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338646#comment-16338646 ] ASF GitHub Bot commented on SOLR-11617: --- Github user dsmiley commented on the issue: https://github.com/apache/lucene-solr/pull/310 Nice docs. Lets not provide an option to configure whitespace/trim handling -- configuration-itis. IMO we should be consistent with other parts of Solr on what to do (e.g. props on core creation) but you seem to feel strongly about it so whatever. It's not that I don't disagree with your point of view, it's only that I think consistency is a higher virtue. Can you add a test please? It could be a simple modification to an existing test. > Expose Alias Metadata CRUD in REST API > -- > > Key: SOLR-11617 > URL: https://issues.apache.org/jira/browse/SOLR-11617 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (8.0) >Reporter: Gus Heck >Priority: Major > Attachments: SOLR_11617.patch, SOLR_11617.patch > > > SOLR-11487 is adding Java API for metadata on aliases, this task is to expose > that functionality to end-users via a REST API. > Some proposed commands, for initial discussion: > - *SETALIASMETA* - upsert, or delete if blank/null/white-space provided. > - *GETALIASMETA* - read existing alias metadata > Given that the parent ticket to this task is going to rely on the alias > metadata, and I suspect a user would potentially completely break their time > partitioned data configuration by editing system metadata directly, we should > either document these commands as "use at your own risk, great > power/responsibility etc" or consider protecting some subset of metadata. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11898) ConcurrentModificationException when calling org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62)
[ https://issues.apache.org/jira/browse/SOLR-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338163#comment-16338163 ] ASF GitHub Bot commented on SOLR-11898: --- GitHub user millerjeff0 opened a pull request: https://github.com/apache/lucene-solr/pull/312 Solr 11898 SOLR-11898 adding classes missed in SOLR-11898 You can merge this pull request into a Git repository by running: $ git pull https://github.com/millerjeff0/lucene-solr SOLR-11898 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/312.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #312 commit af2f43f3d182f46b890db467257b6480b842a16c Author: Jeff Date: 2018-01-24T18:55:24Z SOLR-11898 Updating classes to use ConcurrentHashMap.newKeySet commit 79ceffd56cad6e34815055f408b03e030e1505c6 Author: Jeff Date: 2018-01-24T20:20:42Z SOLR-11898 > ConcurrentModificationException when calling > org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62) > --- > > Key: SOLR-11898 > URL: https://issues.apache.org/jira/browse/SOLR-11898 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Affects Versions: 7.1 >Reporter: Jeff Miller >Priority: Major > Labels: newbie > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > SOLR-11255 Updated to use ConcurrentHashMap.newKeySet(); for all metricNames > implementation but missed a few classes. > I see this exception happening once in a while: > Cause0: java.util.ConcurrentModificationException Cause0-StackTrace: at > java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1558) at > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > at > org.apache.solr.util.stats.MetricUtils.convertMetrics(MetricUtils.java:248) > at > org.apache.solr.util.stats.MetricUtils.convertMetrics(MetricUtils.java:226) > at org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62) > > Three classes weren't updated from using a HashSet for metricNames: > FastLRUCache, LFUCache and LRUCache -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11873) Use time based expiration cache in all places in HdfsDirectoryFactory
[
https://issues.apache.org/jira/browse/SOLR-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336971#comment-16336971
]
ASF GitHub Bot commented on SOLR-11873:
---
GitHub user misutoth opened a pull request:
https://github.com/apache/lucene-solr/pull/311
SOLR-11873: Use a time based expiration cache for one off HDFS FileSy…
…stem instances in all functions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/misutoth/lucene-solr master-hdfsfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/311.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #311
commit 53ba8bd68f025aa786c40e407b29f354bd12e7f5
Author: Mihaly Toth
Date: 2018-01-23T18:58:11Z
SOLR-11873: Use a time based expiration cache for one off HDFS FileSystem
instances in all functions.
> Use time based expiration cache in all places in HdfsDirectoryFactory
> -
>
> Key: SOLR-11873
> URL: https://issues.apache.org/jira/browse/SOLR-11873
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: hdfs
>Affects Versions: 7.2
>Reporter: Mihaly Toth
>Priority: Major
>
> {{HdfsDirectoryFactory.exists()}} method already applies caching on
> FileSystem objects. This is not done yet in the {{size()}} method.
> This function is eventually used when querying the core status. Each and
> every query will use the same configuration and start from the first
> configured HDFS NameNode. If that is down Solr will always access this down
> node first without "learning".
> It would be nice to apply the same caching on that function too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11617) Expose Alias Metadata CRUD in REST API
[ https://issues.apache.org/jira/browse/SOLR-11617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336864#comment-16336864 ] ASF GitHub Bot commented on SOLR-11617: --- GitHub user fsparv opened a pull request: https://github.com/apache/lucene-solr/pull/310 SOLR-11617 Changes as mentioned in Jira comment, plus rather than decide I provided an option for the user to enable whitespace values. By default this is off and whitespace values are treated as null, but if the user really wants to set a value to 3 spaces they can by passing allowWhitespaceValues=true. I'm fairly set against pure whitespace keys unless someone can supply up with a good reason for that. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nsoft/lucene-solr SOLR-11617 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/310.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #310 commit 959401b9ac6bac5e4d696a750c3e5eb42b7da9f5 Author: gus Date: 2018-01-23T19:28:11Z SOLR 11617 - patch 2 commit 218890d7b8c01bddabd1de2a6bbe2f1735ac238c Author: Gus Heck Date: 2018-01-23T20:43:54Z Merge branch 'master' into SOLR-11617 commit 4b4c656a107eba82ec1bfe11e8f5b14b1ea751bf Author: Gus Heck Date: 2018-01-24T03:44:46Z adjust for package changes in master > Expose Alias Metadata CRUD in REST API > -- > > Key: SOLR-11617 > URL: https://issues.apache.org/jira/browse/SOLR-11617 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (8.0) >Reporter: Gus Heck >Priority: Major > Attachments: SOLR_11617.patch, SOLR_11617.patch > > > SOLR-11487 is adding Java API for metadata on aliases, this task is to expose > that functionality to end-users via a REST API. > Some proposed commands, for initial discussion: > - *SETALIASMETA* - upsert, or delete if blank/null/white-space provided. > - *GETALIASMETA* - read existing alias metadata > Given that the parent ticket to this task is going to rely on the alias > metadata, and I suspect a user would potentially completely break their time > partitioned data configuration by editing system metadata directly, we should > either document these commands as "use at your own risk, great > power/responsibility etc" or consider protecting some subset of metadata. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11692) SolrDispatchFilter.closeShield passes the shielded response object back to jetty making the stream unclose able
[ https://issues.apache.org/jira/browse/SOLR-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335373#comment-16335373 ] ASF GitHub Bot commented on SOLR-11692: --- Github user tflobbe commented on the issue: https://github.com/apache/lucene-solr/pull/281 This PR was already merged (SOLR-11692) but wasn't mentioned in the commit message. Could you close @millerjeff0 ? > SolrDispatchFilter.closeShield passes the shielded response object back to > jetty making the stream unclose able > --- > > Key: SOLR-11692 > URL: https://issues.apache.org/jira/browse/SOLR-11692 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Response Writers >Affects Versions: 7.1 > Environment: Linux/Mac tested >Reporter: Jeff Miller >Assignee: David Smiley >Priority: Minor > Labels: dispatchlayer, jetty, newbie, streams > Fix For: 7.3 > > Attachments: SOLR-11692.patch, SOLR-11692.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > In test mode we trigger closeShield code in SolrDispatchFilter, however there > are code paths where we passthrough the objects to the DefaultHandler which > can no longer close the response. > Example stack trace: > java.lang.AssertionError: Attempted close of response output stream. > at > org.apache.solr.servlet.SolrDispatchFilter$2$1.close(SolrDispatchFilter.java:528) > at org.eclipse.jetty.server.Dispatcher.commitResponse(Dispatcher.java:315) > at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:279) > at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:103) > at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:566) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1448) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:385) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326) > at > searchserver.filter.SfdcDispatchFilter.doFilter(SfdcDispatchFilter.java:204) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:370) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:745) > Related JIRA: SOLR-8933 -- This message was sent by At
[jira] [Commented] (SOLR-11459) AddUpdateCommand#prevVersion is not cleared which may lead to problem for in-place updates of non existed documents
[
https://issues.apache.org/jira/browse/SOLR-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328178#comment-16328178
]
ASF GitHub Bot commented on SOLR-11459:
---
GitHub user werder06 opened a pull request:
https://github.com/apache/lucene-solr/pull/307
SOLR-11459 Clear AddUpdateCommand#prevVersion to fix in-place updates…
… for non existed documents
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/werder06/lucene-solr master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/307.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #307
commit 38a689190e80860e9efcfc6bc121f86656801044
Author: Andrey
Date: 2018-01-17T02:17:54Z
SOLR-11459 Clear AddUpdateCommand#prevVersion to fix in-place updates for
non existed documents
> AddUpdateCommand#prevVersion is not cleared which may lead to problem for
> in-place updates of non existed documents
> ---
>
> Key: SOLR-11459
> URL: https://issues.apache.org/jira/browse/SOLR-11459
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 7.0
>Reporter: Andrey Kudryavtsev
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
>
> I have a 1_shard / *m*_replicas SolrCloud cluster with Solr 6.6.0 and run
> batches of 5 - 10k in-place updates from time to time.
> Once I noticed that job "hangs" - it started and couldn't finish for a a
> while.
> Logs were full of messages like:
> {code} Missing update, on which current in-place update depends on, hasn't
> arrived. id=__, looking for version=___, last found version=0" {code}
> {code}
> Tried to fetch document ___ from the leader, but the leader says document has
> been deleted. Deleting the document here and skipping this update: Last found
> version: 0, was looking for: ___",24,0,"but the leader says document has been
> deleted. Deleting the document here and skipping this update: Last found
> version: 0
> {code}
> Further analysis shows that:
> * There are 100-500 updates for non-existed documents among other updates
> (something that I have to deal with)
> * Leader receives bunch of updates and executes this updates one by one.
> {{JavabinLoader}} which is used by processing documents reuses same instance
> of {{AddUpdateCommand}} for every update and just [clearing its state at the
> end|https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java#L99].
> Field [AddUpdateCommand#prevVersion|
> https://github.com/apache/lucene-solr/blob/6396cb759f8c799f381b0730636fa412761030ce/solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java#L76]
> is not cleared.
> * In case of update is in-place update, but specified document does not
> exist, this update is processed as a regular atomic update (i.e. new doc is
> created), but {{prevVersion}} is used as a {{distrib.inplace.prevversion}}
> parameter in sequential calls to every slave in DistributedUpdateProcessor.
> {{prevVersion}} wasn't cleared, so it may contain version from previous
> processed update.
> * Slaves checks it's own version of documents which is 0 (cause doc does not
> exist), slave thinks that some updates were missed and spends 5 seconds in
> [DistributedUpdateProcessor#waitForDependentUpdates|https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java#L99]
> waiting for missed updates (no luck) and also tries to get "correct" version
> from leader (no luck as well)
> * So update for non existed document costs *m* * 5 sec each
> I workarounded this by explicit check of doc existence, but it probably
> should be fixed.
> Obviously first guess is that prevVersion should be cleared in
> {{AddUpdateCommand#clear}}, but have no clue how to test it.
> {code}
> +++ solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java
> (revision )
> @@ -78,6 +78,7 @@
> updateTerm = null;
> isLastDocInBatch = false;
> version = 0;
> + prevVersion = -1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11853) Solr installer: Make check for used tool "service" compatible with SuSE Linux like OpenSuse or SLES
[ https://issues.apache.org/jira/browse/SOLR-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325426#comment-16325426 ] ASF GitHub Bot commented on SOLR-11853: --- GitHub user Mandalka opened a pull request: https://github.com/apache/lucene-solr/pull/305 SOLR-11853: Make check for used tool "service" compatible with SuSE Linux On current SuSE Linux releases like SLES or OpenSuSE the Solr installer stops with the error message "Script requires the 'service' command". This happens because before installation the installer checks if the used command "service" exists by its option "service --version". The command line option "--version" doesn't exist for "service" on current SuSE Linux stable releases. Since the command "service" is there and has an option "--help", this option can be used as additional fallback. So in the pull request i extended the check with "service --help" as second check / fallback before printing this error and exiting. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Mandalka/lucene-solr master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #305 commit 0aa573740b55027192440fc5997a93945ca9b9b3 Author: Markus Mandalka Date: 2018-01-14T02:14:47Z Make check for used tool 'service' compatible with SuSE Linux like OpenSuse or SLES > Solr installer: Make check for used tool "service" compatible with SuSE Linux > like OpenSuse or SLES > --- > > Key: SOLR-11853 > URL: https://issues.apache.org/jira/browse/SOLR-11853 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.2 > Environment: SuSE Linux (SLES & OpenSuSE) >Reporter: Markus Mandalka >Priority: Minor > Labels: github-import, newbie > Fix For: master (8.0), 7.3, 7.2.2 > > > On current SuSE Linux releases like SLES or OpenSuSE the Solr installer stops > with the error message "Script requires the 'service' command". > This happens because before installation the installer checks if the used > command "service" exists by its option "service --version". > The command line option "--version" doesn't exist for "service" on current > SuSE Linux stable releases. > Since the command "service" is there and has an option "--help", this option > can be used as additional fallback. > So in the pull request i extended the check with "service --help" as second > check / fallback before printing this error and exiting. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11624) collection creation should not also overwrite/delete any configset but it can!
[ https://issues.apache.org/jira/browse/SOLR-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325318#comment-16325318 ] ASF GitHub Bot commented on SOLR-11624: --- Github user abhidemon closed the pull request at: https://github.com/apache/lucene-solr/pull/297 > collection creation should not also overwrite/delete any configset but it can! > -- > > Key: SOLR-11624 > URL: https://issues.apache.org/jira/browse/SOLR-11624 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.2 >Reporter: Erick Erickson >Assignee: Ishan Chattopadhyaya > Attachments: SOLR-11624-2.patch, SOLR-11624.3.patch, > SOLR-11624.4.patch, SOLR-11624.patch, SOLR-11624.patch > > > Looks like a problem that crept in when we changed the _default configset > stuff. > setup: > upload a configset named "wiki" > collections?action=CREATE&name=wiki&. > My custom configset "wiki" gets overwritten by _default and then used by the > "wiki" collection. > Assigning to myself only because it really needs to be fixed IMO and I don't > want to lose track of it. Anyone else please feel free to take it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324348#comment-16324348
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user nsoft commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161294218
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+ // We need to add this temporary name just to pass validation.
--- End diff --
ah that's actually checked here:
https://github.com/apache/lucene-solr/pull/304/files#diff-3fe6a8aeb14a57e63507fa17f8346771R207,
but It could be moved to this class (or done both places)
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323976#comment-16323976
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161226045
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+ // We need to add this temporary name just to pass validation.
--- End diff --
I wasn't clear; I don't have any issue with this part of the patch. I'm
suggesting _adding_ a check that we require the configSet. I needed to click
on some line to insert my comment; maybe I should have put my review comment on
the blank line.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323968#comment-16323968
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161223342
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
public class CreateAliasCmd implements Cmd {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String ROUTING_TYPE = "router.name";
+ public static final String ROUTING_FIELD = "router.field";
+ public static final String ROUTING_INCREMENT = "router.interval";
+ public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+ public static final String START = "start";
+ // Collection constants should all reflect names in the v2 structured
input for this command, not v1
+ // names used for CREATE
+ public static final String CREATE_COLLECTION_CONFIG =
"create-collection.config";
--- End diff --
My concern isn't just about the duplication of the Strings/names, it's
about the "set" of them here. Even if we could refer to a constant in some
other class accessible to SolrJ, it would still be a maintenance burden to
refer to the set of those that exist since it's redundant with the collection
creation code. So can we eliminate this entirely (I'd love that) or failing
that have a single place where the set of these strings are defined so we don't
even need to list them in CreateAliasCmd.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323657#comment-16323657
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155289
--- Diff:
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
---
@@ -99,6 +101,8 @@
.parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
.parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
.parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+ // Setting a timezone here is fine as a default, but generally need
to clone with user's timezone, so that
+ // truncation of milliseconds is consistent.
--- End diff --
removed in update
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323659#comment-16323659
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155141
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." +
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+ }
+
+ private Instant validateStart(TimeZone zone, DateTimeFormatter fmt,
String start) {
--- End diff --
checking the ms was the main point, the format and reparse was meant to
trap any timezone/formatter craziness, which mostly goes away with the
collection creation. Simplified, in update
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323649#comment-16323649
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155221
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
--- End diff --
I think I was finding that in running the tests it wasn't always showing
up. I can take it out. Removed here and in cmd class in update
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323656#comment-16323656
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155263
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -962,6 +982,76 @@ public static CollectionOperation get(CollectionAction
action) {
}
}
+ private static SolrParams convertToV1WhenRequired(SolrQueryRequest req,
ModifiableSolrParams params) {
+SolrParams v1Params = params; // (maybe...)
+
+// in the v2 world we get a data map based on the json request, from
the CommandOperation associated
+// with the request, so locate that if we can.. if we find it we have
to translate the v2 request
+// properties to v1 params, otherwise we're already good to go.
+List cmds = req.getCommands(true);
+if (cmds.size() > 1) {
+ // todo: not sure if this is the right thing to do here, but also
not sure what to do if there is more than one...
+ throw new SolrException(BAD_REQUEST, "Only one command is allowed
when creating a routed alias");
+}
+CommandOperation c = cmds.size() == 0 ? null : cmds.get(0);
+if (c != null) { // v2 api, do conversion to v1
+ v1Params = new
BaseHandlerApiSupport.V2ToV1SolrParams(CollectionApiMapping.Meta.CREATE_COLLECTION,
+ req.getPathTemplateValues(), true, params,
+ new CommandOperation("create",
c.getDataMap().get("create-collection")));
+}
+return v1Params;
+ }
+
+ private static Map
parseColletionCreationProps(CollectionsHandler h, SolrParams params, String
prefix)
--- End diff --
fixed in update
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323658#comment-16323658
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155153
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." +
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+ }
+
+ private Instant validateStart(TimeZone zone, DateTimeFormatter fmt,
String start) {
+// This is the normal/easy case, if we can get away with this great!
+TemporalAccessor startTime = attemptTimeStampParsing(start,
zone.toZoneId());
+if (startTime == null) {
+ // No luck, they gave us either date math, or garbage, so we have to
do more work to figure out which and
+ // to make sure it's valid date math and that it doesn't encode any
millisecond precision.
+ ZonedDateTime now =
ZonedDateTime.now(zone.toZoneId()).truncatedTo(ChronoUnit.MILLIS);
+ try {
+Date date = DateMathParser.parseMath(Date.from(now.toInstant()),
start);
+String reformatted =
fmt.format(date.toInstant().truncatedTo(ChronoUnit.MILLIS));
+Date reparse =
Date.from(Instant.from(DATE_TIME_FORMATTER.parse(reformatted)));
+if (!reparse.equals(date)) {
+ throw new SolrException(BAD_REQUEST,
+ "Formatted time did not have the same milliseconds as
original: " + date.getTime() + " vs. " +
+ reparse.getTime() + " This indicates that you used date
math that includes milliseconds. " +
+ "(Hint: 'NOW' used without rounding always has this
problem)" );
+}
+return date.toInstant();
+ } catch (SolrException e) {
+throw new SolrException(BAD_REQUEST,
+"Start Time for the first collection must be a timestamp of
the format -MM-dd_HH_mm_ss, " +
+"or a valid date math expression not containing specific
milliseconds", e);
+ }
+}
+return Instant.from(startTime);
+ }
+
+ private TemporalAccessor attemptTimeStampParsing(String start, ZoneId
zone) {
+try {
+ DATE_TIME_FORMATTER.withZone(zone);
+ return DATE_TIME_FORMATTER.parse(start);
+} catch (DateTimeParseException e) {
+ return null;
+}
+ }
+
+ private boolean anyRoutingParams(ZkNodeProps message) {
+
+return message.containsKey(ROUTING_FIELD) ||
message.containsKey(ROUTING_TYPE) || message.containsKey(START)
+|| message.containsKey(ROUTING_INCREMENT) ||
message.containsKey(TZ);
+ }
+
private void validateAllCollectionsExistAndNoDups(List
collectionList, ZkStateReader zkStateReader) {
final String collectionStr = StrUtils.join(collectionList, ',');
if (new HashSet<>(collectionList).size() != collectionList.size()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(BAD_REQUEST,
String.format(Locale.ROOT, "Can't create collection alias for
collections='%s', since it contains duplicates", collectionStr));
}
ClusterState clusterState = zkStateReader.getClusterState();
Set aliasNames =
zkStateReader.getAliases().getCollectionAliasListMap().keySet();
for (String collection : collectionList) {
if (clusterState.getCollectionOrNull(collection) == null &&
!aliasNames.contains(collection)) {
-throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+throw new SolrException(BAD_REQUEST,
String.format(Locale.ROOT, "Can't create collection alias for
collections='%s', '%s' is not an existing collection o
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323650#comment-16323650
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155243
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+ // We need to add this temporary name just to pass validation.
--- End diff --
The name on this line that I am TMP-ing isn't the collection config name,
it's the name of the collection. We don't want them to pass it but the
collection param parsing code I'm reusing will barf (rightly) if there's no
name for a collection. We'll fix that up later when we calculate the name for
the initial collection in the Cmd class.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323653#comment-16323653
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155184
--- Diff:
solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
---
@@ -218,6 +218,7 @@ public OverseerCollectionMessageHandler(ZkStateReader
zkStateReader, String myId
.put(RELOAD, this::reloadCollection)
.put(DELETE, new DeleteCollectionCmd(this))
.put(CREATEALIAS, new CreateAliasCmd(this))
+.put(CREATEROUTEDALIAS, new CreateAliasCmd(this))
--- End diff --
Seemed like a good idea at the time :) As you like it.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323654#comment-16323654
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161155122
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --
Made my own constants & made set existing constants were too
scattered/disorganized/nonexistant ... fixed in update
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323531#comment-16323531
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161138197
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
An offline chat reveals that I misunderstood the intent some time back.
Names for collections ARE meant to be in UTC, that will make things simpler I
think.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of comma
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323502#comment-16323502
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161134845
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
> you want TZ to adjust any dates that don't include a timezone, and not
adjust any that do (where the only acceptable date included TZ is utc via the Z
suffix)?
Not really because Solr doesn't accept ambiguous dates (dates without a
'Z', == UTC) so that's kind of a non-issue. _The TimeZone is needed to
correctly interpret date math (if present)_. Oh yeah, DateRangeField can
accept very dates without 'Z' but lets ignore that :-) Nonetheless DRF is
strictly UTC.
The format of the collection name is an internal detail so it almost
doesn't matter. I'd rather it be permanently UTC interpreted to avoid problems
in changing TZ somehow. But I can sympathize that there's an argument to be
made it would look nicer if it honored the TZ. But again it's internal.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-1
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323386#comment-16323386
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161122011
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
In reading the issue referenced in the above referenced comment again, it
occurs to me that what I hadn't thought about is that by having TZ parameter at
the top level it does sort of imply that it applies to the whole request, and
we need to handle all dates supplied either as the specified UTC or if the Z is
omitted handle them as the timezone specified. I had mostly been thinking in
terms of the effect of TZ on updates and users trying to optimize by querying
directly to the correct collection and the calculation of "next" collections.
In theory, there could be 2 TZ's one for the routing configuration router.tz
and one that applies to the dates in this request (the top level TZ)... but
that is probably be overkill. So now I think I understand what you've been
trying to tell me... you want TZ to adjust any dates that don't include a
timezone, and not adjust any that do (where the only acceptable date included
TZ is
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323256#comment-16323256
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161110384
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
(and that format has no timzeone in it so in theory we could treat it as
local time based on TZ but at the moment I treat it as utc).
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we a
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323235#comment-16323235
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161107544
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --
yeah, this is stuff that the collection create code adds. I'll look if I
can use constants from there and make a set out of them.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323186#comment-16323186
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161103620
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
actually amending that we do accept the format used for the collection
names, but since a time stamp is valid date math we need to handle this too.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323171#comment-16323171
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161101824
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
So if someone specifies start as an iso instant, we don't want to adjust
that to match the specified zone? This would lead to
start=2018-01-14T20:00:00:00.Z&TZ=EST&router.interval=+1DAY it would seem
that the first bucket should be 2018-01-15 but this way we would be creating a
first bucket for 2018-01-14 instead. That seems confusing. Solr fairly
consistently takes the position that times you speak to it must be in UTC,
changing that here seems awkward. We could allow more general time format and
take the timezone from the entered time, so that they can enter
2018-01-15T00:00:00:00.EST which is unlike the rest of how solr handles
time AFAIK but would make the correspondence between the collection name and
the time value entered intuitive (and maybe complicates combining this with the
datemath)
I had originally proposed that collections names for this be UTC and making
them full time stamps to avoid
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323124#comment-16323124
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161096191
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
--- End diff --
in this area of the code I implement checks for the following:
name
start
router.field
router.name
router.interval
router.max-future-ms
TZ
collection-create.config
Only that last one could possibly be left to the collection-create
infrastructure. The rest are ours (note that the router ones are not the
collection level routing but the alias level routing).
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323095#comment-16323095
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user fsparv commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161093191
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
public class CreateAliasCmd implements Cmd {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String ROUTING_TYPE = "router.name";
+ public static final String ROUTING_FIELD = "router.field";
+ public static final String ROUTING_INCREMENT = "router.interval";
+ public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+ public static final String START = "start";
+ // Collection constants should all reflect names in the v2 structured
input for this command, not v1
+ // names used for CREATE
+ public static final String CREATE_COLLECTION_CONFIG =
"create-collection.config";
--- End diff --
I agree, I don't like having them here and in the collection admin request
but solrj can't see core. Is there a constants class I can move them to that
solrj can see?
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322997#comment-16322997
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161074349
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+ // We need to add this temporary name just to pass validation.
--- End diff --
We need to insist that the collection config name is passed here. It's
very problematic to depend on automatic creation. On a related note, when it's
time to document TRA's, we may want to warn against data driven schema.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322994#comment-16322994
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161077363
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
--- End diff --
I think much of these param validation checks are supposed to be handled at
the CollectionsHandler level; no? I'm not 100% sure.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323000#comment-16323000
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161078867
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." +
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+ }
+
+ private Instant validateStart(TimeZone zone, DateTimeFormatter fmt,
String start) {
--- End diff --
This appears redundant with using DateMathParser earlier and I'm sure we
don't need so much code for this matter (ignore millisecond truncation; in
practice the user is going to do something like `/MONTH` or `/DAY` for start.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322995#comment-16322995
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161076165
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
+
+ // check that the increment is valid date math
+ String checkIncrement = ISO_INSTANT.format(Instant.now()) +
increment;
+ DateMathParser.parseMath(new Date(), checkIncrement); // exception
if invalid increment
--- End diff --
Maybe I wasn't clear before. I mean create a new DateMathParser using the
constructor that accept the timezone (and we can supply ours; null is okay).
Then call the instance method `parseMath(str)` on it (not the static with
'now'). the arg should be the start metadata. The "now" will default to the
current instant in time which is what we want.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. I
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322993#comment-16322993
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161071581
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -962,6 +982,76 @@ public static CollectionOperation get(CollectionAction
action) {
}
}
+ private static SolrParams convertToV1WhenRequired(SolrQueryRequest req,
ModifiableSolrParams params) {
+SolrParams v1Params = params; // (maybe...)
+
+// in the v2 world we get a data map based on the json request, from
the CommandOperation associated
+// with the request, so locate that if we can.. if we find it we have
to translate the v2 request
+// properties to v1 params, otherwise we're already good to go.
+List cmds = req.getCommands(true);
+if (cmds.size() > 1) {
+ // todo: not sure if this is the right thing to do here, but also
not sure what to do if there is more than one...
+ throw new SolrException(BAD_REQUEST, "Only one command is allowed
when creating a routed alias");
+}
+CommandOperation c = cmds.size() == 0 ? null : cmds.get(0);
+if (c != null) { // v2 api, do conversion to v1
+ v1Params = new
BaseHandlerApiSupport.V2ToV1SolrParams(CollectionApiMapping.Meta.CREATE_COLLECTION,
+ req.getPathTemplateValues(), true, params,
+ new CommandOperation("create",
c.getDataMap().get("create-collection")));
+}
+return v1Params;
+ }
+
+ private static Map
parseColletionCreationProps(CollectionsHandler h, SolrParams params, String
prefix)
--- End diff --
typo
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322999#comment-16322999
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161075027
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+ }
+ TimeZone zone;
+ if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+ } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+ }
+ DateTimeFormatter fmt =
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --
No we don't want to use the TZ to format the collection name
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * al
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322992#comment-16322992
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161074912
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
+ throw new NumberFormatException("Negative value not allowed
here");
+}
+ } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+ }
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+ // Validate we got everything we need
+ if (routedField == null || routingType == null || start == null ||
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + "
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+
+ if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+ }
+ // Check for invalid timezone
+ if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
--- End diff --
You needn't explicitly validate TZ here; call TimeZoneUtils.parseTimezone
instead which does so (which I added recently).
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core proper
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322998#comment-16322998
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161079495
--- Diff:
solr/core/src/test/org/apache/solr/cloud/CreateRoutedAliasTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoUnit;
+import java.util.Date;
+import java.util.Locale;
+import java.util.Map;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.HttpEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.InputStreamEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.lucene.util.IOUtils;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.Aliases;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Direct http tests of the CreateRoutedAlias functionality.
+ */
+public class CreateRoutedAliasTest extends SolrCloudTestCase {
--- End diff --
Hmm; it'd be nice to combine this with the existing test to just have one
TimeRoutedAliases test. It can wait.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322996#comment-16322996
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161079147
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." +
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+ }
+
+ private Instant validateStart(TimeZone zone, DateTimeFormatter fmt,
String start) {
+// This is the normal/easy case, if we can get away with this great!
+TemporalAccessor startTime = attemptTimeStampParsing(start,
zone.toZoneId());
+if (startTime == null) {
+ // No luck, they gave us either date math, or garbage, so we have to
do more work to figure out which and
+ // to make sure it's valid date math and that it doesn't encode any
millisecond precision.
+ ZonedDateTime now =
ZonedDateTime.now(zone.toZoneId()).truncatedTo(ChronoUnit.MILLIS);
+ try {
+Date date = DateMathParser.parseMath(Date.from(now.toInstant()),
start);
+String reformatted =
fmt.format(date.toInstant().truncatedTo(ChronoUnit.MILLIS));
+Date reparse =
Date.from(Instant.from(DATE_TIME_FORMATTER.parse(reformatted)));
+if (!reparse.equals(date)) {
+ throw new SolrException(BAD_REQUEST,
+ "Formatted time did not have the same milliseconds as
original: " + date.getTime() + " vs. " +
+ reparse.getTime() + " This indicates that you used date
math that includes milliseconds. " +
+ "(Hint: 'NOW' used without rounding always has this
problem)" );
+}
+return date.toInstant();
+ } catch (SolrException e) {
+throw new SolrException(BAD_REQUEST,
+"Start Time for the first collection must be a timestamp of
the format -MM-dd_HH_mm_ss, " +
+"or a valid date math expression not containing specific
milliseconds", e);
+ }
+}
+return Instant.from(startTime);
+ }
+
+ private TemporalAccessor attemptTimeStampParsing(String start, ZoneId
zone) {
+try {
+ DATE_TIME_FORMATTER.withZone(zone);
+ return DATE_TIME_FORMATTER.parse(start);
+} catch (DateTimeParseException e) {
+ return null;
+}
+ }
+
+ private boolean anyRoutingParams(ZkNodeProps message) {
+
+return message.containsKey(ROUTING_FIELD) ||
message.containsKey(ROUTING_TYPE) || message.containsKey(START)
+|| message.containsKey(ROUTING_INCREMENT) ||
message.containsKey(TZ);
+ }
+
private void validateAllCollectionsExistAndNoDups(List
collectionList, ZkStateReader zkStateReader) {
final String collectionStr = StrUtils.join(collectionList, ',');
if (new HashSet<>(collectionList).size() != collectionList.size()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(BAD_REQUEST,
String.format(Locale.ROOT, "Can't create collection alias for
collections='%s', since it contains duplicates", collectionStr));
}
ClusterState clusterState = zkStateReader.getClusterState();
Set aliasNames =
zkStateReader.getAliases().getCollectionAliasListMap().keySet();
for (String collection : collectionList) {
if (clusterState.getCollectionOrNull(collection) == null &&
!aliasNames.contains(collection)) {
-throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+throw new SolrException(BAD_REQUEST,
String.format(Locale.ROOT, "Can't create collection alias for
collections='%s', '%s' is not an existing collection
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322991#comment-16322991
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161078056
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps
message, NamedList results)
Thread.sleep(100);
}
+ private Map buildAliasMap(String routedField, String
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --
These 3 constants here are unfortunate maybe they could be in a
constant Set and checked here?
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322990#comment-16322990
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161074624
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler
ocmh) {
public void call(ClusterState state, ZkNodeProps message, NamedList
results)
throws Exception {
final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
-
ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+ final List canonicalCollectionList =
parseCollectionsParameter(message.get("collections"));
+ final String canonicalCollectionsString =
StrUtils.join(canonicalCollectionList, ',');
+ validateAllCollectionsExistAndNoDups(canonicalCollectionList,
zkStateReader);
+ holder.applyModificationAndExportToZk(aliases ->
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+ final String routedField = message.getStr(ROUTING_FIELD);
+ final String routingType = message.getStr(ROUTING_TYPE);
+ final String tz = message.getStr(TZ);
+ final String start = message.getStr(START);
+ final String increment = message.getStr(ROUTING_INCREMENT);
+ final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+ try {
+if (0 > Long.valueOf(maxFutureMs)) {
--- End diff --
use parseLong
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322989#comment-16322989
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161071261
--- Diff:
solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
---
@@ -218,6 +218,7 @@ public OverseerCollectionMessageHandler(ZkStateReader
zkStateReader, String myId
.put(RELOAD, this::reloadCollection)
.put(DELETE, new DeleteCollectionCmd(this))
.put(CREATEALIAS, new CreateAliasCmd(this))
+.put(CREATEROUTEDALIAS, new CreateAliasCmd(this))
--- End diff --
Hmm; is there a point to this? I guess it's debatable.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322988#comment-16322988
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161071815
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
public class CreateAliasCmd implements Cmd {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ public static final String ROUTING_TYPE = "router.name";
+ public static final String ROUTING_FIELD = "router.field";
+ public static final String ROUTING_INCREMENT = "router.interval";
+ public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+ public static final String START = "start";
+ // Collection constants should all reflect names in the v2 structured
input for this command, not v1
+ // names used for CREATE
+ public static final String CREATE_COLLECTION_CONFIG =
"create-collection.config";
--- End diff --
Ugh, all those CREATE_COLLECTION_* constants are a maintenance burden;
surely we can avoid them? I really want to avoid double-specificity of the
collection-creation stuff in any shape or form (not in docs/constants/etc.)
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322987#comment-16322987
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161070349
--- Diff:
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
---
@@ -99,6 +101,8 @@
.parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
.parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
.parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+ // Setting a timezone here is fine as a default, but generally need
to clone with user's timezone, so that
+ // truncation of milliseconds is consistent.
--- End diff --
As I indicated in a previous comment on JIRA, this comment isn't true. We
deliberately make the formatted collection name *insensitive* to the timezone.
Heck we ought to say that here.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322985#comment-16322985
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161068781
--- Diff:
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void
addStatusToResponse(NamedList results, RequestStatus
SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
return req.getParams().required().getAll(null, NAME, "collections");
}),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ Map params = req.getParams().required()
+ .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new
String[REQUIRED_ROUTING_PARAMS.size()]));
+ req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+ // subset the params to reuse the collection creation/parsing code
+ ModifiableSolrParams collectionParams =
extractPrefixedParams("create-collection.", req.getParams());
+ if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST,
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
--- End diff --
I believe the general practice for BAD_REQUEST param validation is for Solr
to not log it. Besides, it's redundant with throwing the exception, and any
exception thrown could be chosen to log or not centrally at some entrypoint
juncture or by a thread handler for uncaught exceptions. So lets not litter
log statements that are superfluous unless there's some interesting/useful
context.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322986#comment-16322986
]
ASF GitHub Bot commented on SOLR-11722:
---
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/304#discussion_r161070053
--- Diff:
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
---
@@ -252,6 +256,7 @@ public void processAdd(AddUpdateCommand cmd) throws
IOException {
/** Computes the timestamp of the next collection given the timestamp of
the one before. */
public static Instant computeNextCollTimestamp(Instant fromTimestamp,
String intervalDateMath, TimeZone intervalTimeZone) {
//TODO overload DateMathParser.parseMath to take tz and "now"
+// GH: I don't think that's necessary, you can set the TZ on now when
you pass it in?
--- End diff --
True that it's not necessary; it's just a proposed convenience.
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection
[
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322880#comment-16322880
]
ASF GitHub Bot commented on SOLR-11722:
---
GitHub user nsoft opened a pull request:
https://github.com/apache/lucene-solr/pull/304
SOLR-11722
patch #3 including docs, test SSL fix and precommit fixes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nsoft/lucene-solr SOLR-11722
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/304.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #304
commit aadae0c0e03bff019eac819919430b7c6eb7aabd
Author: Gus Heck
Date: 2018-01-11T20:09:25Z
SOLR-11722 patch #3 including docs, test SSL fix and precommit fixes
> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed
> Alias" along with its first collection. Need to decide what endpoint URL it
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or
> alternatively piggy-back off of command=create-alias but we add more options,
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias
> name via a core property), and then the alias pointing to it which demands
> collections exist first. When figuring the collection name, you'll need to
> reference the format in TimeRoutedAliasUpdateProcessor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321726#comment-16321726
]
ASF GitHub Bot commented on LUCENE-8126:
Github user iverase commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160868387
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320916#comment-16320916
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160769685
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320915#comment-16320915
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160769274
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320917#comment-16320917
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160773587
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx,
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels,
S2Projections.MAX_WIDTH.getClosestLevel(dist *
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) *
DistanceUtils.RADIANS_TO_DEGREES;
+}
+
+@Override
+public Cell getWorldCell() {
+return new S2PrefixTreeCell(this, null);
+}
+
+@Override
+public Cell readCell(BytesRef term, Cell scratch) {
+S2PrefixTreeCell cell = (S2PrefixTreeCell) scratch;
+if (cell == null)
+cell = (S2PrefixTreeCell) getWorldCell();
+cell.readCell(this, term);
+return cell;
+}
+
+@Override
+public CellIterator getTreeCellIterator(Shape shape, int detailLevel) {
+if (!(shape instanceof Point)) {
+return super.getTreeCellIterator(shape, detailLevel);
+}
+Point p = (Point) shape;
+S2CellId id = S2CellId.fromLatLng(S2LatLng.fromDegrees(p.getY(),
p.getX())).parent(detailLevel-1);
+List cells = new ArrayList<>(detailLevel);
+for (int i=0; i < detailLevel -1; i++) {
--- End diff --
nitpick: put a space after that minus operator
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320909#comment-16320909
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160766117
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320913#comment-16320913
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160768405
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320914#comment-16320914
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160773175
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx,
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels,
S2Projections.MAX_WIDTH.getClosestLevel(dist *
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) *
DistanceUtils.RADIANS_TO_DEGREES;
--- End diff --
nitpick: put space after that minus operator
> Spatial prefix tree based on S2 geometry
>
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very
> promising results, in particular for complex shapes (e.g polygons). Using
> this pixelization scheme reduces the size of the index, improves the
> performance of the queries and reduces the loading time for non-point shapes.
> If you are ok with this contribution and before providing any code I would
> like to understand what is the correct/prefered approach:
> 1) Add new depency to the S2 library
> (https://mvnrepository.com/artifact/
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320912#comment-16320912
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160768053
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
--- End diff --
Since this map has a small set of fixed values that have numeric
equivalents, perhaps we can do direct addressing into an array?
> Spatial prefix tree based on S2 geometry
>
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very
> promising results, in particular for complex shapes (e.g polygons). Using
> this pixelization scheme reduces the size of the index, improves the
> performance of the queries and reduces the loading time for non-point shapes.
> If you are ok with this contribution and before providing any code I would
> like to understand what is the correct/prefered approach:
> 1) Add new depency to the S2 library
> (https://mvnrepository.com/artifact/io.sgr/s2-geometry-library-java). It has
> Apache 2.0 license so it should be ok.
> 2) Create a utility class with all methods necessary to navigate the S2 tree
> and create shapes from S2 cells (basically port what we need from the library
> into Lucene).
> What do you think?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320910#comment-16320910
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160768230
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx,
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels,
S2Projections.MAX_WIDTH.getClosestLevel(dist *
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) *
DistanceUtils.RADIANS_TO_DEGREES;
+}
+
+@Override
+public Cell getWorldCell() {
+return new S2PrefixTreeCell(this, null);
+}
+
+@Override
+public Cell readCell(BytesRef term, Cell scratch) {
+S2PrefixTreeCell cell = (S2PrefixTreeCell) scratch;
+if (cell == null)
+cell = (S2PrefixTreeCell) getWorldCell();
--- End diff --
nitpick: our code style in Lucene/Solr is to always use braces
> Spatial prefix tree based on S2 geometry
>
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very
> promising results, in particular for complex
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320911#comment-16320911
]
ASF GitHub Bot commented on LUCENE-8126:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/302#discussion_r160766597
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result =
[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry
[ https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320448#comment-16320448 ] ASF GitHub Bot commented on LUCENE-8126: GitHub user iverase opened a pull request: https://github.com/apache/lucene-solr/pull/302 LUCENE-8126: Spatial prefix tree based on S2 geometry You can merge this pull request into a Git repository by running: $ git pull https://github.com/iverase/lucene-solr master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #302 commit ed57d35c3896c61b8e7af31bce66650800225a34 Author: ivera Date: 2018-01-10T15:29:40Z LUCENE-8126: first commit of S2 RPT commit 9d00f003a0c980ca9eccaff8d4b2e9eebf1e77e2 Author: ivera Date: 2018-01-10T15:31:39Z LUCENE-8126: first commit of S2 RPT commit 6aa0199f07f7c8e23be8ff550fc3fd0aefc5554a Author: ivera Date: 2018-01-10T15:33:36Z LUCENE-8126: Performance test classes. They are included to show the increae of performance using the new RPT. > Spatial prefix tree based on S2 geometry > > > Key: LUCENE-8126 > URL: https://issues.apache.org/jira/browse/LUCENE-8126 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/spatial-extras >Reporter: Ignacio Vera > > Hi [~dsmiley], > I have been working on a prefix tree based on goggle S2 geometry > (https://s2geometry.io/) to be used mainly with Geo3d shapes with very > promising results, in particular for complex shapes (e.g polygons). Using > this pixelization scheme reduces the size of the index, improves the > performance of the queries and reduces the loading time for non-point shapes. > If you are ok with this contribution and before providing any code I would > like to understand what is the correct/prefered approach: > 1) Add new depency to the S2 library > (https://mvnrepository.com/artifact/io.sgr/s2-geometry-library-java). It has > Apache 2.0 license so it should be ok. > 2) Create a utility class with all methods necessary to navigate the S2 tree > and create shapes from S2 cells (basically port what we need from the library > into Lucene). > What do you think? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[ https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320139#comment-16320139 ] ASF GitHub Bot commented on LUCENE-8086: Github user iverase closed the pull request at: https://github.com/apache/lucene-solr/pull/288 > G3d wrapper: Improve circles for non spherical planets > -- > > Key: LUCENE-8086 > URL: https://issues.apache.org/jira/browse/LUCENE-8086 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: Ignacio Vera >Assignee: David Smiley > Fix For: 7.3 > > > Hi [~dsmiley], > The purpose of this ticket is to add a new circle shape (GeoExactCircle) for > non-spherical planets and therefore remove the method relate from > Geo3dCircleShape. The patch will include some simplifications on the wrapper > and some refactoring of the tests. > I will open shortly a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3089) Make ResponseBuilder.isDistrib public
[
https://issues.apache.org/jira/browse/SOLR-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319073#comment-16319073
]
ASF GitHub Bot commented on SOLR-3089:
--
Github user abhidemon closed the pull request at:
https://github.com/apache/lucene-solr/pull/301
> Make ResponseBuilder.isDistrib public
> -
>
> Key: SOLR-3089
> URL: https://issues.apache.org/jira/browse/SOLR-3089
> Project: Solr
> Issue Type: Improvement
> Components: Response Writers
>Affects Versions: 4.0-ALPHA
>Reporter: Rok Rejc
> Fix For: 4.9, 6.0
>
> Attachments: Solr-3089.patch
>
>
> Hi,
> i have posted this issue on a mailing list but i didn't get any response.
> I am trying to write distributed search component (a class that extends
> SearchComponent). I have checked FacetComponent and TermsComponent. If I want
> that search component works in a distributed environment I have to set
> ResponseBuilder's isDistrib to true, like this (this is also done in
> TermsComponent for example):
> public void prepare(ResponseBuilder rb) throws IOException {
> SolrParams params = rb.req.getParams();
> String shards = params.get(ShardParams.SHARDS);
> if (shards != null) {
> List lst = StrUtils.splitSmart(shards, ",",
> true);
> rb.shards = lst.toArray(new String[lst.size()]);
> rb.isDistrib = true;
> }
> }
> If I have my component outside the package org.apache.solr.handler.component
> this doesn't work. Is it possible to make isDistrib public (or is this the
> wrong procedure/behaviour/design)?
> Many thanks,
> Rok
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3089) Make ResponseBuilder.isDistrib public
[
https://issues.apache.org/jira/browse/SOLR-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319070#comment-16319070
]
ASF GitHub Bot commented on SOLR-3089:
--
GitHub user abhidemon opened a pull request:
https://github.com/apache/lucene-solr/pull/301
test: Add test case for ResponseBuilder.isDistributed()
This is being done for https://issues.apache.org/jira/browse/SOLR-3089
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/abhidemon/lucene-solr SOLR-3089
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/301.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #301
commit 767bfbcb143a26bf325a00888c350d06a2486662
Author: abhidemon
Date: 2018-01-09T20:14:34Z
test: Add test case for ResponseBuilder.isDistributed
> Make ResponseBuilder.isDistrib public
> -
>
> Key: SOLR-3089
> URL: https://issues.apache.org/jira/browse/SOLR-3089
> Project: Solr
> Issue Type: Improvement
> Components: Response Writers
>Affects Versions: 4.0-ALPHA
>Reporter: Rok Rejc
> Fix For: 4.9, 6.0
>
> Attachments: Solr-3089.patch
>
>
> Hi,
> i have posted this issue on a mailing list but i didn't get any response.
> I am trying to write distributed search component (a class that extends
> SearchComponent). I have checked FacetComponent and TermsComponent. If I want
> that search component works in a distributed environment I have to set
> ResponseBuilder's isDistrib to true, like this (this is also done in
> TermsComponent for example):
> public void prepare(ResponseBuilder rb) throws IOException {
> SolrParams params = rb.req.getParams();
> String shards = params.get(ShardParams.SHARDS);
> if (shards != null) {
> List lst = StrUtils.splitSmart(shards, ",",
> true);
> rb.shards = lst.toArray(new String[lst.size()]);
> rb.isDistrib = true;
> }
> }
> If I have my component outside the package org.apache.solr.handler.component
> this doesn't work. Is it possible to make isDistrib public (or is this the
> wrong procedure/behaviour/design)?
> Many thanks,
> Rok
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
[
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16318432#comment-16318432
]
ASF GitHub Bot commented on LUCENE-8075:
Github user asfgit closed the pull request at:
https://github.com/apache/lucene-solr/pull/286
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
> Labels: easyfix
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107: fstReader = null; // fr.index is Known NULL here.
> } else {
> fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]);
> // fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group,
> TCA, Institute of Software, Chinese Academy of Sciences" in the
> acknowledgement if applicable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11831) Skip second grouping step if group.limit is 1 (aka Las Vegas patch)
[
https://issues.apache.org/jira/browse/SOLR-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316636#comment-16316636
]
ASF GitHub Bot commented on SOLR-11831:
---
GitHub user mjosephidou opened a pull request:
https://github.com/apache/lucene-solr/pull/300
SOLR-11831: Skip second grouping step if group.limit is 1 (aka Las Vegas
Patch)
Summary:
In cases where we do grouping and ask for {{group.limit=1}} only it is
possible to skip the second grouping step. In our test datasets it improved
speed by around 40%.
Essentially, in the first grouping step each shard returns the top K groups
based on the highest scoring document in each group. The top K groups from each
shard are merged in the federator and in the second step we ask all the shards
to return the top documents from each of the top ranking groups.
If we only want to return the highest scoring document per group we can
return the top document id in the first step, merge results in the federator to
retain the top K groups and then skip the second grouping step entirely.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bloomberg/lucene-solr SOLR-11831
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/300.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #300
commit 6b918c86cd0f37320c32eb669eca722a9e74f768
Author: Malvina Josephidou
Date: 2018-01-04T15:00:35Z
SOLR-11831: Skip second grouping step if group.limit is 1 (aka Las Vegas
patch)
Summary:
In cases where we do grouping and ask for {{group.limit=1}} only it is
possible to skip the second grouping step. In our test datasets it improved
speed by around 40%.
Essentially, in the first grouping step each shard returns the top K groups
based on the highest scoring document in each group. The top K groups from each
shard are merged in the federator and in the second step we ask all the shards
to return the top documents from each of the top ranking groups.
If we only want to return the highest scoring document per group we can
return the top document id in the first step, merge results in the federator to
retain the top K groups and then skip the second grouping step entirely.
> Skip second grouping step if group.limit is 1 (aka Las Vegas patch)
>
>
> Key: SOLR-11831
> URL: https://issues.apache.org/jira/browse/SOLR-11831
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Malvina Josephidou
>Priority: Minor
>
> In cases where we do grouping and ask for {{group.limit=1}} only it is
> possible to skip the second grouping step. In our test datasets it improved
> speed by around 40%.
> Essentially, in the first grouping step each shard returns the top K groups
> based on the highest scoring document in each group. The top K groups from
> each shard are merged in the federator and in the second step we ask all the
> shards to return the top documents from each of the top ranking groups.
> If we only want to return the highest scoring document per group we can
> return the top document id in the first step, merge results in the federator
> to retain the top K groups and then skip the second grouping step entirely.
> This is possible provided that:
> a) We do not need to know the total number of matching documents per group
> b) Within group sort and between group sort is the same.
> c) We are not doing reranking (this is because this is done in the second
> grouping step. It is also possible to get this to work with reranking but
> more work and some additional assumptions are required)
>
> This patch applies the grouping optimisation in cases where a)-c) apply and
> we are only sorting by relevance. It is also possible to extend this work to
> handle multiple sorting criteria and also reranking.
> P.S. Diego and I called this patch "las vegas" because we started to write it
> on the flight to Las Vegas for Lucene/Solr revolution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
[
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314891#comment-16314891
]
ASF GitHub Bot commented on LUCENE-8075:
Github user imgpulak commented on the issue:
https://github.com/apache/lucene-solr/pull/286
@ jpountz @mikemccand I have updated as per your comment. Please review the
same.
Thanks !
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
> Labels: easyfix
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107: fstReader = null; // fr.index is Known NULL here.
> } else {
> fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]);
> // fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group,
> TCA, Institute of Software, Chinese Academy of Sciences" in the
> acknowledgement if applicable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11484) CloudSolrClient's cache of collection clusterstate can cause RouteExceptions when attempting directUpdates after collection modifications
[
https://issues.apache.org/jira/browse/SOLR-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307051#comment-16307051
]
ASF GitHub Bot commented on SOLR-11484:
---
Github user SerCeMan commented on the issue:
https://github.com/apache/lucene-solr/pull/264
Closing as this might be fixed by
https://issues.apache.org/jira/browse/SOLR-11484
> CloudSolrClient's cache of collection clusterstate can cause RouteExceptions
> when attempting directUpdates after collection modifications
> -
>
> Key: SOLR-11484
> URL: https://issues.apache.org/jira/browse/SOLR-11484
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Hoss Man
>Assignee: Noble Paul
> Fix For: 7.2, master (8.0)
>
> Attachments: SOLR-11484.patch, SOLR-11484.patch,
> jenkins.thetaphi.20662.txt
>
>
> This was discovered while auditing jenkins failures from
> {{TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete}}
> (where a test explicitly deletes and then recreates a collection with the
> same name), but as noted in a comment below, SOLR-11392 is another example of
> non-obvious test failures that can pop up because of this bug.
> In practice, it can affect any CloudSolrClient user after changes have been
> made to a collection (to add/move replicas, etc...)
>
> Original jira notes...
> {{TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete}}
> seems to fail with non-trivial frequency, so I grabbed the logs from a recent
> failure and starting trying to follow along with the actions to figure out
> what exactly is happening
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20662/
> {noformat}
>[junit4] ERROR 20.3s J1 |
> TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete <<<
>[junit4]> Throwable #1:
> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from
> server at https://127.0.0.1:42959/solr/testcollection_shard1_replica_n3:
> Expected mime type a
> pplication/octet-stream but got text/html.
>[junit4]>
>[junit4]> content="text/html;charset=ISO-8859-1"/>
>[junit4]> Error 404
> {noformat}
> The crux of this failure appears to be a genuine bug in how CloudSolrClient
> uses it's cached ClusterState info when doing (direct) updates. The key bits
> seem to be:
> * CloudSolrClient does _something_ (update,query,etc...) with a collection
> causing the current cluster state for the collection to be cached
> * The actual collection changes such that a Solr node/core no longer exists
> as part of the collection
> * CloudSolrClient is asked to process an UpdateRequest which triggers the
> code paths for the {{directUpdate()}} method -- which attempts to route the
> updates directly to a replica of the appropriate shard using the (cache)
> collection state info
> * CloudSolrClient (may) attempt to send that UpdateRequest to a node/core
> that doesn't exist, getting a 404 -- which does not (seem to) trigger a state
> refresh, or retry to find a correct URL to resend the update to.
> Details to follow in comment
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11624) collection creation should not also overwrite/delete any configset but it can!
[ https://issues.apache.org/jira/browse/SOLR-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306661#comment-16306661 ] ASF GitHub Bot commented on SOLR-11624: --- GitHub user abhidemon opened a pull request: https://github.com/apache/lucene-solr/pull/297 fix: https://issues.apache.org/jira/browse/SOLR-11624 You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhidemon/lucene-solr SOLR-11624 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/297.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #297 commit 01eac2e19d0badf095816fa8f8a546653c0e4556 Author: abhidemon Date: 2017-12-12T20:44:56Z fix: https://issues.apache.org/jira/browse/SOLR-11624 > collection creation should not also overwrite/delete any configset but it can! > -- > > Key: SOLR-11624 > URL: https://issues.apache.org/jira/browse/SOLR-11624 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.2 >Reporter: Erick Erickson >Assignee: Ishan Chattopadhyaya > Attachments: SOLR-11624-2.patch, SOLR-11624.3.patch, > SOLR-11624.4.patch, SOLR-11624.patch > > > Looks like a problem that crept in when we changed the _default configset > stuff. > setup: > upload a configset named "wiki" > collections?action=CREATE&name=wiki&. > My custom configset "wiki" gets overwritten by _default and then used by the > "wiki" collection. > Assigning to myself only because it really needs to be fixed IMO and I don't > want to lose track of it. Anyone else please feel free to take it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11804) Test RankQuery in distributed mode
[
https://issues.apache.org/jira/browse/SOLR-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306291#comment-16306291
]
ASF GitHub Bot commented on SOLR-11804:
---
GitHub user diegoceccarelli opened a pull request:
https://github.com/apache/lucene-solr/pull/296
SOLR-11804: Test RankQuery in distributed mode
Currently `RankQuery` is not tested in distributed mode. I added a few
tests in `TestDistributedSearch` to check that it works properly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bloomberg/lucene-solr SOLR-11804
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/296.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #296
commit 1591ec8e3c5e0cd670890642b34aba4eb22e0a99
Author: Diego Ceccarelli
Date: 2017-12-29T13:51:04Z
SOLR-11804: Test RankQuery in distributed mode
> Test RankQuery in distributed mode
> --
>
> Key: SOLR-11804
> URL: https://issues.apache.org/jira/browse/SOLR-11804
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> Currently {{RankQuery}} is not tested in distributed mode. I added a few
> tests in `TestDistributedSearch` to check that it works properly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11800) Improve error message when parsing RankQuery
[
https://issues.apache.org/jira/browse/SOLR-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305583#comment-16305583
]
ASF GitHub Bot commented on SOLR-11800:
---
Github user diegoceccarelli commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/295#discussion_r158969387
--- Diff: solr/core/src/java/org/apache/solr/search/QParser.java ---
@@ -360,6 +360,10 @@ public static QParser getParser(String qstr, String
parserName, boolean allowLoc
}
QParserPlugin qplug = req.getCore().getQueryPlugin(parserName);
+if (qplug == null){
+ // error: log ?
--- End diff --
I would log an error here, but `QParser` doesn't have a logger, should we
add one?
> Improve error message when parsing RankQuery
>
>
> Key: SOLR-11800
> URL: https://issues.apache.org/jira/browse/SOLR-11800
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> When a user specifies something wrong for the parameter {{rq}} sometimes it
> is hard to understand where is the problem, this patch attempts to improve
> the error message returned in the response.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11800) Improve error message when parsing RankQuery
[ https://issues.apache.org/jira/browse/SOLR-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305580#comment-16305580 ] ASF GitHub Bot commented on SOLR-11800: --- GitHub user diegoceccarelli opened a pull request: https://github.com/apache/lucene-solr/pull/295 SOLR-11800: Improve error message when parsing RankQuery When a user specifies something wrong for the parameter `rq` sometimes it is hard to understand where is the problem, this patch attempts to improve the error message returned in the response. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bloomberg/lucene-solr SOLR-11800 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/295.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #295 commit 64928e43710eb86be0f7e23adbc8107da124efb8 Author: Diego Ceccarelli Date: 2017-12-28T16:22:55Z SOLR-11800: Improve error message when parsing RankQuery > Improve error message when parsing RankQuery > > > Key: SOLR-11800 > URL: https://issues.apache.org/jira/browse/SOLR-11800 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Diego Ceccarelli >Priority: Minor > > when a user specify something wrong for the parameter rq sometimes it is hard > to understand where is the problem, this patch attempts to improve the error > message returned in the response. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11622) Bundled mime4j library not sufficient for Tika requirement
[
https://issues.apache.org/jira/browse/SOLR-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304573#comment-16304573
]
ASF GitHub Bot commented on SOLR-11622:
---
Github user mrkarthik closed the pull request at:
https://github.com/apache/lucene-solr/pull/273
> Bundled mime4j library not sufficient for Tika requirement
> --
>
> Key: SOLR-11622
> URL: https://issues.apache.org/jira/browse/SOLR-11622
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Affects Versions: 7.1, 6.6.2
>Reporter: Karim Malhas
>Assignee: Karthik Ramachandran
>Priority: Minor
> Labels: build
> Fix For: 7.3
>
> Attachments: SOLR-11622.patch, SOLR-11622.patch
>
>
> The version 7.2 of Apache James Mime4j bundled with the Solr binary releases
> does not match what is required by Apache Tika for parsing rfc2822 messages.
> The master branch for james-mime4j seems to contain the missing Builder class
> [https://github.com/apache/james-mime4j/blob/master/core/src/main/java/org/apache/james/mime4j/stream/MimeConfig.java
> ]
> This prevents import of rfc2822 formatted messages. For example like so:
> {{./bin/post -c dovecot -type 'message/rfc822' 'testdata/email_01.txt'
> }}
> And results in the following stacktrace:
> java.lang.NoClassDefFoundError:
> org/apache/james/mime4j/stream/MimeConfig$Builder
> at
> org.apache.tika.parser.mail.RFC822Parser.parse(RFC822Parser.java:63)
> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> at
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:135)
> at
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:228)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.jav
[jira] [Commented] (SOLR-11620) Occasional NullPointerException in CloudSolrClient
[
https://issues.apache.org/jira/browse/SOLR-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299301#comment-16299301
]
ASF GitHub Bot commented on SOLR-11620:
---
Github user tflobbe commented on the issue:
https://github.com/apache/lucene-solr/pull/264
I believe this is the same as reported in SOLR-11620
> Occasional NullPointerException in CloudSolrClient
> ---
>
> Key: SOLR-11620
> URL: https://issues.apache.org/jira/browse/SOLR-11620
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 6.6.1
>Reporter: Rob Trickey
>Priority: Minor
>
> When sending document updates to Solr, we occasionally see the following
> error:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1182)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
> {noformat}
> Looking at the code, there is a lack of null check on requestedCollections
> around the for loop
> for (DocCollection ext : requestedCollections)
> which causes the error. Wrapping this loop with if(requestedCollections !=
> null) would solve the problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection
[ https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299071#comment-16299071 ] ASF GitHub Bot commented on SOLR-8327: -- Github user slackhappy commented on the issue: https://github.com/apache/lucene-solr/pull/294 I updated my PR to target SOLR-8327 more specifically, and got the tests to pass. I think a smarter approach like that used by CloudSolrClient would be great. My understanding of the change in SOLR-10524 is that even the smartest/fastest updates of zookeeper data won't match the real-world state of the cluster in many situations, such as replica state changes, because those will be batched, but certainly a smarter approach would narrow that gap as much as possible, in addition to reducing the amount of state fetching. > SolrDispatchFilter is not caching new state format, which results in live > fetch from ZK per request if node does not contain core from collection > - > > Key: SOLR-8327 > URL: https://issues.apache.org/jira/browse/SOLR-8327 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 5.3 >Reporter: Jessica Cheng Mallet >Assignee: Varun Thacker > Labels: solrcloud > Attachments: SOLR-8327.patch > > > While perf testing with non-solrj client (request can be sent to any solr > node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G > for 20 second dump). From the thread dump, we noticed this: > java.lang.Object.wait (Native Method) > java.lang.Object.wait (Object.java:503) > org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309) > org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation > (ZkCmdExecutor.java:61) > org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkStateReader.getCollectionLive > (ZkStateReader.java:841) > org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515) > org.apache.solr.common.cloud.ClusterState.getCollectionOrNull > (ClusterState.java:175) > org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98) > org.apache.solr.servlet.HttpSolrCall.getCoreByCollection > (HttpSolrCall.java:784) > org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272) > org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:210) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:179) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1652) > org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) > org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) > org.eclipse.jetty.server.session.SessionHandler.doHandle > (SessionHandler.java:223) > org.eclipse.jetty.server.handler.ContextHandler.doHandle > (ContextHandler.java:1127) > org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) > org.eclipse.jetty.server.session.SessionHandler.doScope > (SessionHandler.java:185) > org.eclipse.jetty.server.handler.ContextHandler.doScope > (ContextHandler.java:1061) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle > (ContextHandlerCollection.java:215) > org.eclipse.jetty.server.handler.HandlerCollection.handle > (HandlerCollection.java:110) > org.eclipse.jetty.server.handler.HandlerWrapper.handle > (HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle (Server.java:499) > org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) > org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) > org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob > (QueuedThreadPool.java:635) > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run > (QueuedThreadPool.java:555) > java.lang.Thread.run (Thread.java:745) > Looks like SolrDispatchFilter doesn't have caching similar to the > collectionStateCache in CloudSolrClient, so if the node doesn't know about a > collection in the new state format, it just live-fetch it from Zookeeper on > every request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) ---
[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection
[ https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297686#comment-16297686 ] ASF GitHub Bot commented on SOLR-8327: -- Github user slackhappy commented on the issue: https://github.com/apache/lucene-solr/pull/294 I'll look into the test failures. I actually didn't mean to create the PR yet 😳 > SolrDispatchFilter is not caching new state format, which results in live > fetch from ZK per request if node does not contain core from collection > - > > Key: SOLR-8327 > URL: https://issues.apache.org/jira/browse/SOLR-8327 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 5.3 >Reporter: Jessica Cheng Mallet >Assignee: Varun Thacker > Labels: solrcloud > Attachments: SOLR-8327.patch > > > While perf testing with non-solrj client (request can be sent to any solr > node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G > for 20 second dump). From the thread dump, we noticed this: > java.lang.Object.wait (Native Method) > java.lang.Object.wait (Object.java:503) > org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309) > org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation > (ZkCmdExecutor.java:61) > org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkStateReader.getCollectionLive > (ZkStateReader.java:841) > org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515) > org.apache.solr.common.cloud.ClusterState.getCollectionOrNull > (ClusterState.java:175) > org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98) > org.apache.solr.servlet.HttpSolrCall.getCoreByCollection > (HttpSolrCall.java:784) > org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272) > org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:210) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:179) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1652) > org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) > org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) > org.eclipse.jetty.server.session.SessionHandler.doHandle > (SessionHandler.java:223) > org.eclipse.jetty.server.handler.ContextHandler.doHandle > (ContextHandler.java:1127) > org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) > org.eclipse.jetty.server.session.SessionHandler.doScope > (SessionHandler.java:185) > org.eclipse.jetty.server.handler.ContextHandler.doScope > (ContextHandler.java:1061) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle > (ContextHandlerCollection.java:215) > org.eclipse.jetty.server.handler.HandlerCollection.handle > (HandlerCollection.java:110) > org.eclipse.jetty.server.handler.HandlerWrapper.handle > (HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle (Server.java:499) > org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) > org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) > org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob > (QueuedThreadPool.java:635) > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run > (QueuedThreadPool.java:555) > java.lang.Thread.run (Thread.java:745) > Looks like SolrDispatchFilter doesn't have caching similar to the > collectionStateCache in CloudSolrClient, so if the node doesn't know about a > collection in the new state format, it just live-fetch it from Zookeeper on > every request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection
[ https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297495#comment-16297495 ] ASF GitHub Bot commented on SOLR-8327: -- Github user chatman commented on the issue: https://github.com/apache/lucene-solr/pull/294 Seems like there are some test failures due to this change: [junit4] Tests with failures [seed: DE1D5337E38D2C32]: [junit4] - org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas [junit4] - org.apache.solr.cloud.TestPullReplica.testAddRemovePullReplica [junit4] - org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas [junit4] - org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest [junit4] - org.apache.solr.cloud.DeleteShardTest.testDirectoryCleanupAfterDeleteShard [junit4] - org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest [junit4] - org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest [junit4] - org.apache.solr.cloud.TestUtilizeNode.test [junit4] - org.apache.solr.cloud.TestTlogReplica.testAddRemoveTlogReplica Haven't looked into them yet, though. > SolrDispatchFilter is not caching new state format, which results in live > fetch from ZK per request if node does not contain core from collection > - > > Key: SOLR-8327 > URL: https://issues.apache.org/jira/browse/SOLR-8327 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 5.3 >Reporter: Jessica Cheng Mallet >Assignee: Varun Thacker > Labels: solrcloud > Attachments: SOLR-8327.patch > > > While perf testing with non-solrj client (request can be sent to any solr > node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G > for 20 second dump). From the thread dump, we noticed this: > java.lang.Object.wait (Native Method) > java.lang.Object.wait (Object.java:503) > org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309) > org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation > (ZkCmdExecutor.java:61) > org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkStateReader.getCollectionLive > (ZkStateReader.java:841) > org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515) > org.apache.solr.common.cloud.ClusterState.getCollectionOrNull > (ClusterState.java:175) > org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98) > org.apache.solr.servlet.HttpSolrCall.getCoreByCollection > (HttpSolrCall.java:784) > org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272) > org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:210) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:179) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1652) > org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) > org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) > org.eclipse.jetty.server.session.SessionHandler.doHandle > (SessionHandler.java:223) > org.eclipse.jetty.server.handler.ContextHandler.doHandle > (ContextHandler.java:1127) > org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) > org.eclipse.jetty.server.session.SessionHandler.doScope > (SessionHandler.java:185) > org.eclipse.jetty.server.handler.ContextHandler.doScope > (ContextHandler.java:1061) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle > (ContextHandlerCollection.java:215) > org.eclipse.jetty.server.handler.HandlerCollection.handle > (HandlerCollection.java:110) > org.eclipse.jetty.server.handler.HandlerWrapper.handle > (HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle (Server.java:499) > org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) > org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) > org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob > (QueuedThreadPool.java:635) > org.eclipse.jetty.util.thread.Queued
[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection
[ https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297410#comment-16297410 ] ASF GitHub Bot commented on SOLR-8327: -- Github user elyograg commented on the issue: https://github.com/apache/lucene-solr/pull/294 Java programs are migrating to nanoTime instead of currentTimeMillis for elapsed time because many people have found that the latter will go *backwards* on occasion. It is not monotonic. Using nanoTime should be far less likely to go backwards. That undesirable behavior has been observed in the wild, but should be rare. Supposedly nanoTime is monotonic if the OS properly supports a monotonic clock. There's a lot of info out there about it: https://www.google.com/search?q=java+nanotime+monotonic The fact that nanoTime *might* produce elapsed times with greater accuracy than one millisecond is a bonus. > SolrDispatchFilter is not caching new state format, which results in live > fetch from ZK per request if node does not contain core from collection > - > > Key: SOLR-8327 > URL: https://issues.apache.org/jira/browse/SOLR-8327 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 5.3 >Reporter: Jessica Cheng Mallet >Assignee: Varun Thacker > Labels: solrcloud > Attachments: SOLR-8327.patch > > > While perf testing with non-solrj client (request can be sent to any solr > node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G > for 20 second dump). From the thread dump, we noticed this: > java.lang.Object.wait (Native Method) > java.lang.Object.wait (Object.java:503) > org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309) > org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation > (ZkCmdExecutor.java:61) > org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkStateReader.getCollectionLive > (ZkStateReader.java:841) > org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515) > org.apache.solr.common.cloud.ClusterState.getCollectionOrNull > (ClusterState.java:175) > org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98) > org.apache.solr.servlet.HttpSolrCall.getCoreByCollection > (HttpSolrCall.java:784) > org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272) > org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:210) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:179) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1652) > org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) > org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) > org.eclipse.jetty.server.session.SessionHandler.doHandle > (SessionHandler.java:223) > org.eclipse.jetty.server.handler.ContextHandler.doHandle > (ContextHandler.java:1127) > org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) > org.eclipse.jetty.server.session.SessionHandler.doScope > (SessionHandler.java:185) > org.eclipse.jetty.server.handler.ContextHandler.doScope > (ContextHandler.java:1061) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle > (ContextHandlerCollection.java:215) > org.eclipse.jetty.server.handler.HandlerCollection.handle > (HandlerCollection.java:110) > org.eclipse.jetty.server.handler.HandlerWrapper.handle > (HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle (Server.java:499) > org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) > org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) > org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob > (QueuedThreadPool.java:635) > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run > (QueuedThreadPool.java:555) > java.lang.Thread.run (Thread.java:745) > Looks like SolrDispatchFilter doesn't have caching similar to the > collectionStateCache in CloudSolrClient, so if the node doesn't know about a > collection in the new state format, it just live-fetch it fr
[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection
[ https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297372#comment-16297372 ] ASF GitHub Bot commented on SOLR-8327: -- Github user dragonsinth commented on the issue: https://github.com/apache/lucene-solr/pull/294 This approach seems fine to me. Remind me why we use nanoTime vs. normal clock? I'm sure you're right I just want to refresh my brain. > SolrDispatchFilter is not caching new state format, which results in live > fetch from ZK per request if node does not contain core from collection > - > > Key: SOLR-8327 > URL: https://issues.apache.org/jira/browse/SOLR-8327 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 5.3 >Reporter: Jessica Cheng Mallet >Assignee: Varun Thacker > Labels: solrcloud > Attachments: SOLR-8327.patch > > > While perf testing with non-solrj client (request can be sent to any solr > node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G > for 20 second dump). From the thread dump, we noticed this: > java.lang.Object.wait (Native Method) > java.lang.Object.wait (Object.java:503) > org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309) > org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345) > org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation > (ZkCmdExecutor.java:61) > org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342) > org.apache.solr.common.cloud.ZkStateReader.getCollectionLive > (ZkStateReader.java:841) > org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515) > org.apache.solr.common.cloud.ClusterState.getCollectionOrNull > (ClusterState.java:175) > org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98) > org.apache.solr.servlet.HttpSolrCall.getCoreByCollection > (HttpSolrCall.java:784) > org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272) > org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:210) > org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:179) > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter > (ServletHandler.java:1652) > org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) > org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) > org.eclipse.jetty.server.session.SessionHandler.doHandle > (SessionHandler.java:223) > org.eclipse.jetty.server.handler.ContextHandler.doHandle > (ContextHandler.java:1127) > org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) > org.eclipse.jetty.server.session.SessionHandler.doScope > (SessionHandler.java:185) > org.eclipse.jetty.server.handler.ContextHandler.doScope > (ContextHandler.java:1061) > org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle > (ContextHandlerCollection.java:215) > org.eclipse.jetty.server.handler.HandlerCollection.handle > (HandlerCollection.java:110) > org.eclipse.jetty.server.handler.HandlerWrapper.handle > (HandlerWrapper.java:97) > org.eclipse.jetty.server.Server.handle (Server.java:499) > org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) > org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) > org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob > (QueuedThreadPool.java:635) > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run > (QueuedThreadPool.java:555) > java.lang.Thread.run (Thread.java:745) > Looks like SolrDispatchFilter doesn't have caching similar to the > collectionStateCache in CloudSolrClient, so if the node doesn't know about a > collection in the new state format, it just live-fetch it from Zookeeper on > every request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching
[ https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297192#comment-16297192 ] ASF GitHub Bot commented on SOLR-10524: --- GitHub user slackhappy opened a pull request: https://github.com/apache/lucene-solr/pull/294 ZkStateReader: cache LazyCollectionRef SOLR-10524 introduced zk state update batching, with a default interval of 2 seconds. That opens the door for a simple, time-based cache on the read side to address the issue described in SOLR-8327 You can merge this pull request into a Git repository by running: $ git pull https://github.com/slackhappy/lucene-solr cloud_cache_lazy_collection_ref Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #294 commit e7c6a6773f1d01d2ddb0dbce6cdbaeff52e78376 Author: John Gallagher Date: 2017-12-19T16:57:25Z ZkStateReader: cache LazyCollectionRef SOLR-10524 introduced zk state update batching, with a default interval of 2 seconds. That opens the door for a simple, time-based cache on the read side to address the issue described in SOLR-8327 > Better ZkStateWriter batching > - > > Key: SOLR-10524 > URL: https://issues.apache.org/jira/browse/SOLR-10524 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Cao Manh Dat > Fix For: 6.6, 7.0 > > Attachments: SOLR-10524-NPE-fix.patch, SOLR-10524-dragonsinth.patch, > SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch > > > There are several JIRAs (I'll link in a second) about trying to be more > efficient about processing overseer messages as the overseer can become a > bottleneck, especially with very large numbers of replicas in a cluster. One > of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read > large no:of items say 1. put them into in memory buckets and feed them > into overseer". > This JIRA is to break out that part of the discussion as it might be an easy > win whereas "eliminating the Overseer queue" would be quite an undertaking. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11701) Upgrade to Tika 1.17 when available
[ https://issues.apache.org/jira/browse/SOLR-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291932#comment-16291932 ] ASF GitHub Bot commented on SOLR-11701: --- GitHub user tballison opened a pull request: https://github.com/apache/lucene-solr/pull/291 jira/solr-11701 SOLR-11701 upgrade to Tika 1.17 You can merge this pull request into a Git repository by running: $ git pull https://github.com/tballison/lucene-solr jira/solr-11701 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #291 commit c5d4e37de782b2491b3e71cfbb004e5022b55f6b Author: Karthik Ramachandran Date: 2017-11-14T00:21:44Z SOLR-11622: Fix mime4j library dependency for Tika commit 40b246b12e8fc6455e023d9d60b8edcfab9b184e Author: Karthik Ramachandran Date: 2017-12-01T22:12:15Z Merge remote-tracking branch 'upstream/master' into jira/solr-11622 commit 21f2ab483f356fad9b89233e544457a07540afd1 Author: Karthik Ramachandran Date: 2017-12-03T03:50:01Z SOLR-11622: Fix bundled mime4j library not sufficient for Tika requirement commit a40ca80ed7036732a332f5508589ae32eb4b Author: Karthik Ramachandran Date: 2017-12-04T15:33:18Z Merge remote-tracking branch 'upstream/master' into jira/solr-11622 commit a0d6fba8c2e85565a02a8565882a627fa7ceccc4 Author: Karthik Ramachandran Date: 2017-12-14T16:24:45Z Merge remote-tracking branch 'upstream/master' into jira/SOLR-11622 commit c2c885f8a2e2c49fab6f737b13f0ff9a1346714c Author: Karthik Ramachandran Date: 2017-12-14T20:45:09Z SOLR-11622: Fix mime4j library dependency for Tika commit e834693a31d0b410a7e0205e1eecda55206a44fa Author: tballison Date: 2017-12-15T02:20:51Z SOLR-11701 - upgrade to Tika 1.17 > Upgrade to Tika 1.17 when available > --- > > Key: SOLR-11701 > URL: https://issues.apache.org/jira/browse/SOLR-11701 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Tim Allison > > Kicking off release process for Tika 1.17 in the next few days. Please let > us know if you have any requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2857) Multi-content-type /update handler
[ https://issues.apache.org/jira/browse/SOLR-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289242#comment-16289242 ] ASF GitHub Bot commented on SOLR-2857: -- GitHub user steeveb972 opened a pull request: https://github.com/apache/lucene-solr/pull/290 Removed deprecated reference XmlUpdateRequestHandler has been replaced by UpdateRequestHandler as of SOLR-2857 You can merge this pull request into a Git repository by running: $ git pull https://github.com/steeveb972/lucene-solr master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #290 commit 43dece91ffdac55344ff2fbf65eb1207dff6bec7 Author: steeveb972 Date: 2017-12-13T13:12:50Z Removed deprecated reference XmlUpdateRequestHandler has been replaced by UpdateRequestHandler > Multi-content-type /update handler > -- > > Key: SOLR-2857 > URL: https://issues.apache.org/jira/browse/SOLR-2857 > Project: Solr > Issue Type: Improvement >Reporter: Erik Hatcher >Assignee: Ryan McKinley > Fix For: 4.0-ALPHA > > Attachments: SOLR-2857-content-type-refactor.patch, > SOLR-2857-content-type-refactor.patch, SOLR-2857-content-type-refactor.patch, > SOLR-2857-update-content-type.patch, SOLR-2857-update-content-type.patch, > SOLR-2857-update-content-type.patch, SOLR-2857_oldUrls.patch > > > Something I've been thinking about lately... it'd be great to get rid of all > the specific update handlers like /update/csv, /update/extract, and > /update/json and collapse them all into a single /update that underneath uses > the content-type(s) to hand off to specific content handlers. This would > make it much easier to toss content at Solr and provide a single entry point > for updates. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8011) Improve similarity explanations
[
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288923#comment-16288923
]
ASF GitHub Bot commented on LUCENE-8011:
Github user jpountz commented on the issue:
https://github.com/apache/lucene-solr/pull/280
Merged, thank you @mayya-sharipova.
> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
> Labels: newdev
> Fix For: master (8.0)
>
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
> 2.2 = scaling factor, k1 + 1
> 9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
> 0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like
> docCount/docFreq without explanation:
> {noformat}
> product of:
> 0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
> 2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more
> practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8011) Improve similarity explanations
[
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288907#comment-16288907
]
ASF GitHub Bot commented on LUCENE-8011:
Github user asfgit closed the pull request at:
https://github.com/apache/lucene-solr/pull/280
> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
> Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
> 2.2 = scaling factor, k1 + 1
> 9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
> 0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like
> docCount/docFreq without explanation:
> {noformat}
> product of:
> 0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
> 2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more
> practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
[
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288541#comment-16288541
]
ASF GitHub Bot commented on LUCENE-8075:
Github user mikemccand commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/286#discussion_r156538059
--- Diff:
lucene/core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
---
@@ -106,37 +106,37 @@ public IntersectTermsEnum(FieldReader fr, Automaton
automaton, RunAutomaton runA
if (fr.index == null) {
fstReader = null;
--- End diff --
Sorry for the slow respons ehere @imgpulak and @jpountz but Adrien is
right: `fr.index` can never be null anymore. So I think we should change the
code to `assert` it's never null and only do the `else` clause of the current
`if` statement?
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
> Labels: easyfix
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107: fstReader = null; // fr.index is Known NULL here.
> } else {
> fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]);
> // fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group,
> TCA, Institute of Software, Chinese Academy of Sciences" in the
> acknowledgement if applicable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8011) Improve similarity explanations
[
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288340#comment-16288340
]
ASF GitHub Bot commented on LUCENE-8011:
Github user jpountz commented on the issue:
https://github.com/apache/lucene-solr/pull/280
No need to be sorry!
> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
> Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
> 2.2 = scaling factor, k1 + 1
> 9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
> 0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like
> docCount/docFreq without explanation:
> {noformat}
> product of:
> 0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
> 2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more
> practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8011) Improve similarity explanations
[
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288338#comment-16288338
]
ASF GitHub Bot commented on LUCENE-8011:
Github user mayya-sharipova commented on the issue:
https://github.com/apache/lucene-solr/pull/280
@jpountz Adrien thanks for your help. Sorry, I will make sure to run `ant
precommit` before committing next time. I have pushed another change to address
this.
> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
> Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
> 2.2 = scaling factor, k1 + 1
> 9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
> 0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like
> docCount/docFreq without explanation:
> {noformat}
> product of:
> 0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
> 2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more
> practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11711) distributed pivot & field facets can processes excessive docs unneccessarily due to internal mincount=0
[
https://issues.apache.org/jira/browse/SOLR-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287988#comment-16287988
]
ASF GitHub Bot commented on SOLR-11711:
---
Github user HoustonPutman closed the pull request at:
https://github.com/apache/lucene-solr/pull/279
> distributed pivot & field facets can processes excessive docs unneccessarily
> due to internal mincount=0
> ---
>
> Key: SOLR-11711
> URL: https://issues.apache.org/jira/browse/SOLR-11711
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: faceting
>Affects Versions: master (8.0)
>Reporter: Houston Putman
>Assignee: Hoss Man
> Labels: pull-request-available
> Fix For: master (8.0), 7.3
>
>
> Currently while sending pivot facet requests to each shard, the
> {{facet.pivot.mincount}} is set to {{0}} if the facet is sorted by count with
> a specified limit > 0. However with a mincount of 0, the pivot facet will use
> exponentially more wasted memory for every pivot field added. This is because
> there will be a total of {{limit^(# of pivots)}} pivot values created in
> memory, even though the vast majority of them will have counts of 0, and are
> therefore useless.
> Imagine the scenario of a pivot facet with 3 levels, and
> {{facet.limit=1000}}. There will be a billion pivot values created, and there
> will almost definitely be nowhere near a billion pivot values with counts > 0.
> This likely due to the reasoning mentioned in [this comment in the original
> distributed pivot facet
> ticket|https://issues.apache.org/jira/browse/SOLR-2894?focusedCommentId=13979898].
> Basically it was thought that the refinement code would need to know that a
> count was 0 for a shard so that a refinement request wasn't sent to that
> shard. However this is checked in the code, [in this part of the refinement
> candidate
> checking|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.1.0/solr/core/src/java/org/apache/solr/handler/component/PivotFacetField.java#L275].
> Therefore if the {{pivot.mincount}} was set to 1, the non-existent values
> would either:
> * Not be known, because the {{facet.limit}} was smaller than the number of
> facet values with positive counts. This isn't an issue, because they wouldn't
> have been returned with {{pivot.mincount}} set to 0.
> * Would be known, because the {{facet.limit}} would be larger than the number
> of facet values returned. therefore this conditional would return false
> (since we are only talking about pivot facets sorted by count).
> The solution, is to use the same pivot mincount as would be used if no limit
> was specified.
> This also relates to a similar problem in field faceting that was "fixed" in
> [SOLR-8988|https://issues.apache.org/jira/browse/SOLR-8988#13324]. The
> solution was to add a flag, {{facet.distrib.mco}}, which would enable not
> choosing a mincount of 0 when unnessesary. Since this flag can only increase
> performance, and doesn't break any queries I have removed it as an option and
> replaced the code to use the feature always.
> There was one code change necessary to fix the MCO option, since the
> refinement candidate selection logic had a bug. The bug only occured with a
> minCount > 0 and limit > 0 specified. When a shard replied with less than the
> limit requested, it would assume the next maximum count on that shard was the
> {{mincount}}, where it would actually be the {{mincount-1}} (because a facet
> value with a count of mincount would have been returned). Therefore the MCO
> didn't cause any errors, but with a mincount of 1 the refinement logic always
> assumed that the shard had more values with a count of 1.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8011) Improve similarity explanations
[
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287881#comment-16287881
]
ASF GitHub Bot commented on LUCENE-8011:
Github user jpountz commented on the issue:
https://github.com/apache/lucene-solr/pull/280
Thanks @mayya-sharipova, this looks great. `ant precommit` complains from
some missing docs (the build requires that all public/protected APIs have some
minimal documentation), could you fix it?
> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
> Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
> 2.2 = scaling factor, k1 + 1
> 9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
> 0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl))
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like
> docCount/docFreq without explanation:
> {noformat}
> product of:
> 0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
> 2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more
> practical.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287651#comment-16287651
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155938328
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
---
@@ -150,10 +176,21 @@ public Rectangle rect(double minX, double maxX,
double minY, double maxY) {
@Override
public Circle circle(double x, double y, double distance) {
-GeoCircle circle = GeoCircleFactory.makeGeoCircle(planetModel,
-y * DistanceUtils.DEGREES_TO_RADIANS,
-x * DistanceUtils.DEGREES_TO_RADIANS,
-distance * DistanceUtils.DEGREES_TO_RADIANS);
+GeoCircle circle;
+if (planetModel.ab == planetModel.c) {
--- End diff --
@DaddyWri comments?
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287650#comment-16287650
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155938315
--- Diff:
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/ShapeRectRelationTestCase.java
---
@@ -28,15 +28,15 @@
import static
org.locationtech.spatial4j.distance.DistanceUtils.DEGREES_TO_RADIANS;
-public abstract class Geo3dShapeRectRelationTestCase extends
RandomizedShapeTestCase {
+public abstract class ShapeRectRelationTestCase extends
RandomizedShapeTestCase {
protected final static double RADIANS_PER_DEGREE = Math.PI/180.0;
@Rule
public final TestLog testLog = TestLog.instance;
protected int maxRadius = 180;
- public Geo3dShapeRectRelationTestCase() {
+ public ShapeRectRelationTestCase() {
super(SpatialContext.GEO);
--- End diff --
I think the subclass should pass in the context.
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11745) SolrCore doesn't log core if too many closes called on it
[
https://issues.apache.org/jira/browse/SOLR-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286633#comment-16286633
]
ASF GitHub Bot commented on SOLR-11745:
---
GitHub user millerjeff0 opened a pull request:
https://github.com/apache/lucene-solr/pull/289
SOLR-11745: change logging references to getName
SOLR-11745
this doesn't resolve to anything by object name, Eg:
org.apache.solr.core.SolrCore@4812a0d7
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/millerjeff0/lucene-solr SOLR-11745
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/289.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #289
commit 6f87979e088b49ccb12d74af9a4352cb920665c9
Author: Jeff
Date: 2017-12-11T21:54:42Z
SOLR-11745: change logging references to getName
> SolrCore doesn't log core if too many closes called on it
> -
>
> Key: SOLR-11745
> URL: https://issues.apache.org/jira/browse/SOLR-11745
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: logging
>Affects Versions: 7.1
>Reporter: Jeff Miller
>Priority: Trivial
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> log.error("Too many close [count:{}] on {}. Please report this
> exception to solr-u...@lucene.apache.org", count, this );
> Calling this just prints
> org.apache.solr.core.SolrCore@4812a0d7
> Suggest changing to getName
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285077#comment-16285077
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155938309
--- Diff:
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/ShapeRectRelationTestCase.java
---
@@ -75,9 +75,9 @@ public void testGeoCircleRect() {
new Geo3dRectIntersectionTestHelper(ctx) {
--- End diff --
Should be renamed as this class is no longer Geo3d specific
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
[
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284972#comment-16284972
]
ASF GitHub Bot commented on LUCENE-8075:
Github user imgpulak commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/286#discussion_r155930642
--- Diff:
lucene/core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
---
@@ -106,37 +106,37 @@ public IntersectTermsEnum(FieldReader fr, Automaton
automaton, RunAutomaton runA
if (fr.index == null) {
fstReader = null;
--- End diff --
@jpountz, @mikemccand any update here?
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
> Labels: easyfix
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Possible null pointer dereference in
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107: fstReader = null; // fr.index is Known NULL here.
> } else {
> fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]);
> // fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group,
> TCA, Institute of Software, Chinese Academy of Sciences" in the
> acknowledgement if applicable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[ https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284004#comment-16284004 ] ASF GitHub Bot commented on LUCENE-8086: Github user iverase commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/288#discussion_r155844958 --- Diff: lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java --- @@ -16,29 +16,15 @@ */ package org.apache.lucene.spatial.spatial4j; -import java.util.ArrayList; -import java.util.List; - -import org.apache.lucene.spatial3d.geom.GeoPath; -import org.apache.lucene.spatial3d.geom.GeoPolygon; +import org.junit.Rule; +import org.junit.Test; import org.locationtech.spatial4j.TestLog; import org.locationtech.spatial4j.context.SpatialContext; -import org.locationtech.spatial4j.distance.DistanceUtils; import org.locationtech.spatial4j.shape.Circle; import org.locationtech.spatial4j.shape.Point; import org.locationtech.spatial4j.shape.RectIntersectionTestHelper; -import org.apache.lucene.spatial3d.geom.LatLonBounds; --- End diff -- I didn't dear to change it but that was the idea of the effort, I will remove Geo3d. > G3d wrapper: Improve circles for non spherical planets > -- > > Key: LUCENE-8086 > URL: https://issues.apache.org/jira/browse/LUCENE-8086 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: Ignacio Vera > > Hi [~dsmiley], > The purpose of this ticket is to add a new circle shape (GeoExactCircle) for > non-spherical planets and therefore remove the method relate from > Geo3dCircleShape. The patch will include some simplifications on the wrapper > and some refactoring of the tests. > I will open shortly a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283996#comment-16283996
]
ASF GitHub Bot commented on LUCENE-8086:
Github user iverase commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155844229
--- Diff:
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
---
@@ -155,16 +107,12 @@ protected Geo3dShape generateRandomShape(Point nearP)
{
ulhcPoint = lrhcPoint;
lrhcPoint = temp;
}
-final GeoBBox shape = GeoBBoxFactory.makeGeoBBox(planetModel,
ulhcPoint.getY() * DEGREES_TO_RADIANS,
-lrhcPoint.getY() * DEGREES_TO_RADIANS,
-ulhcPoint.getX() * DEGREES_TO_RADIANS,
-lrhcPoint.getX() * DEGREES_TO_RADIANS);
-return new Geo3dShape(shape, ctx);
+return (Geo3dShape) ctx.getShapeFactory().rect(lrhcPoint,
ulhcPoint);
--- End diff --
Indeed!
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283991#comment-16283991
]
ASF GitHub Bot commented on LUCENE-8086:
Github user iverase commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155843938
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
---
@@ -150,10 +176,21 @@ public Rectangle rect(double minX, double maxX,
double minY, double maxY) {
@Override
public Circle circle(double x, double y, double distance) {
-GeoCircle circle = GeoCircleFactory.makeGeoCircle(planetModel,
-y * DistanceUtils.DEGREES_TO_RADIANS,
-x * DistanceUtils.DEGREES_TO_RADIANS,
-distance * DistanceUtils.DEGREES_TO_RADIANS);
+GeoCircle circle;
+if (planetModel.ab == planetModel.c) {
--- End diff --
I think spatial3d is a low level library in that respect, so it shouldn't
have such a method. Karl Wright has the last word, comment would be fine
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[ https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283993#comment-16283993 ] ASF GitHub Bot commented on LUCENE-8086: Github user iverase commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/288#discussion_r155844074 --- Diff: lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java --- @@ -55,6 +55,13 @@ private SpatialContext context; private PlanetModel planetModel; + /** + * Default accuracy for circles when not using the unit sphere. + * It is equivalent to 10m on the surface of the earth. + */ + private static double DEFAULT_CIRCLE_ACCURACY = 1.6e-6; --- End diff -- Indeed! > G3d wrapper: Improve circles for non spherical planets > -- > > Key: LUCENE-8086 > URL: https://issues.apache.org/jira/browse/LUCENE-8086 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: Ignacio Vera > > Hi [~dsmiley], > The purpose of this ticket is to add a new circle shape (GeoExactCircle) for > non-spherical planets and therefore remove the method relate from > Geo3dCircleShape. The patch will include some simplifications on the wrapper > and some refactoring of the tests. > I will open shortly a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283985#comment-16283985
]
ASF GitHub Bot commented on LUCENE-8086:
Github user iverase commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155843390
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
---
@@ -67,6 +74,25 @@ public SpatialContext getSpatialContext() {
return context;
}
+ /**
+ * Set the accuracy for circles.
+ *
+ * "Accuracy" is defined as the maximum linear distance between any
point on the
+ * surface circle and planes that describe the circle. Therefore on
WSG84, since the
+ * radius of earth is 6,371,000 meters, an accuracy of 1e-6 corresponds
to 6.3 meters.
+ * For an accuracy of 1.0 meters, the value of 1.6e-7.
+ *
+ * The default value is set to 10m (1.6e-6).
+ *
+ * Note that accuracy has no effect when the planet model is a sphere.
In that case circles
+ * are always fully precise.
+ *
+ * @param circleAccuracy the provided accuracy as a linear distance.
--- End diff --
I need to ask Karl Wright if that is what it means, but I guess so. I will
update accordingly.
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283854#comment-16283854
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155822251
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
---
@@ -67,6 +74,25 @@ public SpatialContext getSpatialContext() {
return context;
}
+ /**
+ * Set the accuracy for circles.
+ *
+ * "Accuracy" is defined as the maximum linear distance between any
point on the
+ * surface circle and planes that describe the circle. Therefore on
WSG84, since the
+ * radius of earth is 6,371,000 meters, an accuracy of 1e-6 corresponds
to 6.3 meters.
+ * For an accuracy of 1.0 meters, the value of 1.6e-7.
+ *
+ * The default value is set to 10m (1.6e-6).
+ *
+ * Note that accuracy has no effect when the planet model is a sphere.
In that case circles
+ * are always fully precise.
+ *
+ * @param circleAccuracy the provided accuracy as a linear distance.
--- End diff --
by "linear distance" do you mean decimal degrees as is used in other parts
of the Spatial4j API? If so please say "decimal degrees". If not, perhaps it
should be in that unit?
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[ https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283853#comment-16283853 ] ASF GitHub Bot commented on LUCENE-8086: Github user dsmiley commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/288#discussion_r155825386 --- Diff: lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java --- @@ -16,29 +16,15 @@ */ package org.apache.lucene.spatial.spatial4j; -import java.util.ArrayList; -import java.util.List; - -import org.apache.lucene.spatial3d.geom.GeoPath; -import org.apache.lucene.spatial3d.geom.GeoPolygon; +import org.junit.Rule; +import org.junit.Test; import org.locationtech.spatial4j.TestLog; import org.locationtech.spatial4j.context.SpatialContext; -import org.locationtech.spatial4j.distance.DistanceUtils; import org.locationtech.spatial4j.shape.Circle; import org.locationtech.spatial4j.shape.Point; import org.locationtech.spatial4j.shape.RectIntersectionTestHelper; -import org.apache.lucene.spatial3d.geom.LatLonBounds; --- End diff -- If I get this right, you've removed the Geo3D dependencies of this test. Yet it's still named to be Geo3d? > G3d wrapper: Improve circles for non spherical planets > -- > > Key: LUCENE-8086 > URL: https://issues.apache.org/jira/browse/LUCENE-8086 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: Ignacio Vera > > Hi [~dsmiley], > The purpose of this ticket is to add a new circle shape (GeoExactCircle) for > non-spherical planets and therefore remove the method relate from > Geo3dCircleShape. The patch will include some simplifications on the wrapper > and some refactoring of the tests. > I will open shortly a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283852#comment-16283852
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155826444
--- Diff:
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dDistanceCalculator.java
---
@@ -73,62 +74,20 @@ public boolean within(Point from, double toX, double
toY, double distance) {
@Override
public Point pointOnBearing(Point from, double distDEG, double
bearingDEG, SpatialContext ctx, Point reuse) {
-// Algorithm using Vincenty's formulae
(https://en.wikipedia.org/wiki/Vincenty%27s_formulae)
-// which takes into account that planets may not be spherical.
-//Code adaptation from
http://www.movable-type.co.uk/scripts/latlong-vincenty.html
Geo3dPointShape geoFrom = (Geo3dPointShape) from;
GeoPoint point = (GeoPoint) geoFrom.shape;
-double lat = point.getLatitude();
-double lon = point.getLongitude();
double dist = DistanceUtils.DEGREES_TO_RADIANS * distDEG;
double bearing = DistanceUtils.DEGREES_TO_RADIANS * bearingDEG;
-
--- End diff --
Yay
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[ https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283848#comment-16283848 ] ASF GitHub Bot commented on LUCENE-8086: Github user dsmiley commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/288#discussion_r155821440 --- Diff: lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java --- @@ -55,6 +55,13 @@ private SpatialContext context; private PlanetModel planetModel; + /** + * Default accuracy for circles when not using the unit sphere. + * It is equivalent to 10m on the surface of the earth. + */ + private static double DEFAULT_CIRCLE_ACCURACY = 1.6e-6; --- End diff -- should be final > G3d wrapper: Improve circles for non spherical planets > -- > > Key: LUCENE-8086 > URL: https://issues.apache.org/jira/browse/LUCENE-8086 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: Ignacio Vera > > Hi [~dsmiley], > The purpose of this ticket is to add a new circle shape (GeoExactCircle) for > non-spherical planets and therefore remove the method relate from > Geo3dCircleShape. The patch will include some simplifications on the wrapper > and some refactoring of the tests. > I will open shortly a pull request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets
[
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283851#comment-16283851
]
ASF GitHub Bot commented on LUCENE-8086:
Github user dsmiley commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/288#discussion_r155824604
--- Diff:
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
---
@@ -155,16 +107,12 @@ protected Geo3dShape generateRandomShape(Point nearP)
{
ulhcPoint = lrhcPoint;
lrhcPoint = temp;
}
-final GeoBBox shape = GeoBBoxFactory.makeGeoBBox(planetModel,
ulhcPoint.getY() * DEGREES_TO_RADIANS,
-lrhcPoint.getY() * DEGREES_TO_RADIANS,
-ulhcPoint.getX() * DEGREES_TO_RADIANS,
-lrhcPoint.getX() * DEGREES_TO_RADIANS);
-return new Geo3dShape(shape, ctx);
+return (Geo3dShape) ctx.getShapeFactory().rect(lrhcPoint,
ulhcPoint);
--- End diff --
change is good but the variable names are wrong. `rect(lowerLeft,
upperRight)`
> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for
> non-spherical planets and therefore remove the method relate from
> Geo3dCircleShape. The patch will include some simplifications on the wrapper
> and some refactoring of the tests.
> I will open shortly a pull request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
