[
https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878979#comment-16878979
]
Gus Heck commented on SOLR-13375:
---------------------------------
First potentially functional patch. Lots of refactoring to move logic into the
routed alias classes from the maintain Cmd classes, which are now consolidated
to a single MaintainRoutedAlias class. Also refactored many longer methods into
smaller chunks. The basic strategy here is to use a specialized subclass of the
primary routed alias classes to provide the context for answering the question
of whether or not collections need to be created. I also altered the basic
logic such that the routed aliases fully calculate the collection to which the
document should be routed and then push that "target" collection to the
maintain command repeatedly until the collection required has been created.
This simplifies the situation which previously had
MaintainCategoryRoutedAliasCmd working off of the value encountered in the
document and the MaintainTimeRoutedAlias was just marching the collections
forward without knowledge of the end-state. The imbalance in those strategies
needed to be resolved to keep DRA's tractable. Another notable abstraction
added is a notion of "actions" that are requested by each routed alias during
the execution of the MaintainRoutedAliasCmd. In the case of DRA's these are
generated by each sub-dimension and collated into a final set of actions. by
the DRA.
Additionally I hit a very time consuming bug with tests, where I eventually
realized that the problem is that the results of an admin command become
visible (to the test) before the execution of the command is entirely
completed, and the test that has waited for a collection to be visible can
begin to shut down while an Async operation is still in progress. This can lead
to never being released from the watcher.await(timeout); call in
OverseerTaskQueue.offer never releasing (and then the shutdown cycle that is
waiting for the core async thread to terminate waits until the timeout
expires). This only showed up if I saturated my CPU and then only ablut 20% of
the time. The sneaky thing about this is if you beasted it and went to bed or
went to lunch it would complete successfully because of the timeout, but the
time it took to do so was ridiculous if you were waiting for it.
A 5 second Thread.sleep() as the last line of the test reliably resolved this,
but not being happy with that I added a count of pending overseerTasks and a
allowOverseerPendingTasksToComplete() method to OverseerTaskQueue and the first
thing that happens on CoreContainer.shutdown is it calls the new method (which
of course first prohibits new tasks from being queued... though I'm not sure if
the exception thrown to threads that try is ideal...). Once the in-progress
tasks finish shutdown proceeds normally. This completely solved the problems
with my async collection creation tests. [^SOLR-13375.patch]
> Dimensional Routed Aliases
> --------------------------
>
> Key: SOLR-13375
> URL: https://issues.apache.org/jira/browse/SOLR-13375
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
> Affects Versions: master (9.0)
> Reporter: Gus Heck
> Assignee: Gus Heck
> Priority: Major
> Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch
>
>
> Current available routed aliases are restricted to a single field. This
> feature will allow Solr to provide data driven collection access, creation
> and management based on multiple fields in a document. The collections will
> be queried and updated in a unified manner via an alias. Current routing is
> restricted to the values of a single field. The particularly useful
> combination at this time will be Category X Time routing but Category X
> Category may also be useful. More importantly, if additional routing schemes
> are created in the future (either as contributions or as custom code by
> users) combination among these should be supported.
> It is expected that not all combinations will be useful, and that
> determination of usefulness I expect to leave up to the user. Some Routing
> schemes may need to be limited to be the leaf/last routing scheme for
> technical reasons, though I'm not entirely convinced of that yet. If so, a
> flag will be added to the RoutedAlias interface.
> Initial desire is to support two levels, though if arbitrary levels can be
> supported easily that will be done.
> This could also have been called CompositeRoutedAlias, but that creates a TLA
> clash with CategoryRoutedAlias.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
