[
https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836080#comment-16836080
]
Tomás Fernández Löbbe commented on SOLR-13439:
----------------------------------------------
None of those cases use the feature as it's supposed to be used. You either
need frequent access to the properties (i.e. per request, timer, something) or
not (i.e. maybe to validate something after a particular type of request, like
a collection API with a particular set of values, etc)
Case 1 (Frequent access):
1. T+0: SolrCore starts (i.e. A replica is added, a collection is created, you
name it)
2. T+0: On initialization, a component that relies on collection properties
registers a listener. Solr reads from ZooKeeper once.
3. T+0 to T+end-of-life-of-the-core: The watch remains.
4. T+end-of-life-of-the-core: no more reads are expected, the watch is removed.
Times read with current approach: 1 + Number of modifications of properties.
Times with the cache approach: unknown, at least the same as with the
listener, but depends on when the properties are accessed.
Case 2: Don't set a watch. (you only do this if you know you are going to
access the properties a low number of times, so low that it doesn't make sense
to spend resources in watching).
T + 0: read the properties (reads from Zookeeper)
done.
{quote}Code that calls getCollectionProperties() is either fast or slow
depending on whether or not someone else has set a watch on that collection.
{quote}
If your component doesn't set a listener, assume it's going to ZooKeeper. You'd
only do this if reads to a property are infrequent enough (see case 2 above)
{quote}I think the current inconsistency is worse because something that was
fast due to the action of unrelated code (getCollectionProperties()) can become
surprisingly slow,
{quote}
Again, assume {{getCollectionProperties()}} goes to ZooKeeper, but this is only
used in case of infrequent access. The caller makes the informed tradeoff of
going to ZooKeeper per call instead of spending resources (ZooKeeper’s and
Solr’s) in watching.
{quote}whereas after my patch, setting the watch may become surprisingly fast
due to the effect of unrelated code.
{quote}
... if it's in cache you mean?
{quote}in no case will it be more
{quote}
I can give you two cases:
Case 1: Collection properties are accessed infrequently (like in my “case 2
above”), but collection properties change frequently (i.e. every second)
1. T + 0: call to getCollectionProperties(), Zk watch is set and element is on
cache
2. T + 1 to T + 9: Collection properties changes, fires watches to Solr. Solr
receives the watch and reads from Zookeeper
3. T + 10 cache expires
With cache, we read from Zookeeper 10 times, and ZooKeeper fires 10 watches.
Without cache, we read once, ZooKeeper doens't fire any watch. Keep in mind
that some clusters may have many collections (hundreds/thousands?), this may
add a lot of load to ZooKeeper for things that aren’t going to be needed.
Case 2: A component doesn’t rely on a listener, but relies on cache.
1. T + 0: call to getCollectionProperties(), Zk watch is set and element is on
cache
2. T + 10, cache expires
3. T + 11: call to getCollectionProperties(), Zk watch is set and element is
on cache
4. T + 20, cache expires
5. …
With a listener, this is just one read. With cache, this is, again, unknown,
but up to N, the number of calls to {{getCollectionProperties()}}
> Make collection properties easier and safer to use in code
> ----------------------------------------------------------
>
> Key: SOLR-13439
> URL: https://issues.apache.org/jira/browse/SOLR-13439
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: master (9.0)
> Reporter: Gus Heck
> Assignee: Gus Heck
> Priority: Major
> Attachments: SOLR-13439.patch, SOLR-13439.patch
>
>
> (breaking this out from SOLR-13420, please read there for further background)
> Before this patch the api is quite confusing (IMHO):
> # any code that wanted to know what the properties for a collection are
> could call zkStateReader.getCollectionProperties(collection) but this was a
> dangerous and trappy API because that was a query to zookeeper every time. If
> a naive user auto-completed that in their IDE without investigating, heavy
> use of zookeeper would ensue.
> # To "do it right" for any code that might get called on a per-doc or per
> request basis one had to cause caching by registering a watcher. At which
> point the getCollectionProperties(collection) magically becomes safe to use,
> but the watcher pattern probably looks famillar induces a user who hasn't
> read the solr code closely to create their own cache and update it when their
> watcher is notified. If the caching side effect of watches isn't understood
> this will lead to many in-memory copies of collection properties maintained
> in user code.
> # This also creates a task to be scheduled on a thread (PropsNotification)
> and induces an extra thread-scheduling lag before the changes can be observed
> by user code.
> # The code that cares about collection properties needs to have a lifecycle
> tied to either a collection or something other object with an even more
> ephemeral life cycle such as an URP. The user now also has to remember to
> ensure the watch is unregistered, or there is a leak.
> After this patch
> # Calls to getCollectionProperties(collection) are always safe to use in any
> code anywhere. Caching and cleanup are automatic.
> # Code that really actually wants to know if a collection property changes
> so it can wake up and do something (autoscaling?) still has the option of
> registering a watcher that will asynchronously send them a notification.
> # Updates can be observed sooner via getCollectionProperties with no need to
> wait for a thread to run. (vs a cache held in user code)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
