Noble: Those sound like some big and bold ideas that are deserving of a separate conversation? My proposal here is based on the APIs we have today.
~ David On Fri, Jul 31, 2020 at 10:06 PM Noble Paul <[email protected]> wrote: > I would also wish to deprecate all the public APIs which uses concrete > classes and move over to interfaces. > > The objectives are. > * We should be able to use POJOs that help with strong typing > * We should be able to easily switch between JSON/javabin/XML or even > well known protocols like protobuf > > On Sat, Aug 1, 2020 at 12:02 PM Noble Paul <[email protected]> wrote: > > > > Thanks David for bringing this up. > > > > I want us to stop using concrete classes. We should exclusively use > > interfaces in our public APIs. > > > > We should stop using NamedList and start using this interface > > > > > https://github.com/apache/lucene-solr/blob/b91b461632b19c9488926a9302126a2158df3298/solr/solrj/src/java/org/apache/solr/common/util/SimpleMap.java > > > > NamedList will implement this interface too > > > > The SolrRequest is an abstract class and it has a huge surface area. > > We must create a simple small interface for this as will > > > > > > > > > > On Sat, Aug 1, 2020 at 6:25 AM David Smiley <[email protected]> wrote: > > > > > > Hello devs, > > > > > > I'd like to have an API design discussion here on the dev list > concerning adding asynchronous request semantics to SolrClient. I think > the SolrClient APIs require great peer-review consideration and so I'm > having this discussion here instead of a JIRA issue. > > > > > > Background: Sometimes, a SolrJ user (including Solr itself) may wish > to issue a request but not block/wait for the response because it can do > other useful work before eventually needing the results. For example, > maybe the user has multiple requests to send to Solr that can be done in > parallel and _then_ process the results. This is in fact what Solr needs > internally for during distributed-search (see HttpShardHandler) and during > distributed-indexing (see ConcurrentUpdateHttp2SolrClient). > > > > > > Why?: Greater efficiency. This can be accomplished easily in a > generic fashion with threads (e.g. > CompletableFuture.supplyAsync(Supplier,Executor)) with use of the > SolrClient concurrently. However, this can result in lots of threads and > context switching. I really like Cao Dat's explanation in the description > here: https://issues.apache.org/jira/browse/SOLR-14354 Thanks to HTTP/2, > there is a new approach in which the underlying HTTP client can handle the > asynchronous nature more efficiently with minimal threads. Complete > plumbing of this implies that SolrClient (and/or its derivatives) need a > similar async mechanism. Disclaimer: I haven't measured any of this but > am piggybacking off of Cao's's insights on SOLR-14354 > > > > > > Where?: An async API _could_ go only on some SolrClient classes that > natively support it, avoiding changing SolrClient itself. Maybe this is > what should occur first before "graduating" the method to SolrClient where > we devise a default approach, although I would prefer just putting it on > SolrClient. The default approach might be configured to throw > UnsupportedOperationException, or perhaps might simply use an Executor to > get it done in an obvious way (assuming we can get ahold of an Executor > somewhere?). If you're writing a delegating SolrClient (which I've done in > the past), you might want to take-care to delegate this method. > > > Another aspect of "Where" is whether SolrRequest should additionally > have an API alternative to "process" which is currently synchronous and > calls out to SolrClient.request(this). > > > > > > What?: What should this API look like? This is my primary interest > right now and the meat of the discussion. > > > > > > SolrClient's primary signature that actually does the work > (synchronously) is this: > > > > > > public abstract NamedList<Object> > request(@SuppressWarnings({"rawtypes"})final SolrRequest request, String > collection) > > > throws SolrServerException, IOException; > > > > > > I don't like the "collection" there as it's redundant with either the > configured SolrClient default or a setting in SolrRequest, but I digress > from the important matter at hand. > > > > > > In SOLR-14354, recently committed by Cao Dat destined for Solr 8.7, > there is a new (undocumented) method on Http2SolrClient (*not* SolrClient): > > > > > > public Cancellable asyncRequest(@SuppressWarnings({"rawtypes"}) > SolrRequest solrRequest, String collection, > AsyncListener<NamedList<Object>> asyncListener) { > > > > > > So firstly, it's only on Http2SolrClient, which means if you're using > CloudHttp2SolrClient (which does not subclass it, counterintuitively to > users) then too bad -- SOLR-14675 filed by my colleague. > > > Secondly, I really dislike this API signature (sorry Cao). It > introduces two custom and obscure Solr interfaces, Cancellable and > AsyncListener. I claim that we can and should use one abstraction provided > by the JDK -- CompletableFuture<NamedList<Object>>. The user can cancel() > it if they like, they can use it like a Future which is maybe what they > prefer, or they can attach a listener / handler via whenComplete(lambda) > that even has access to an exception if there was one. Based on some > toying around with CompletableFuture lately, I suspect what would be most > straight-forward is to have an asyncRequest method that *returns* a > CompletableFuture, and which merely takes the SolrRequest parameter and > nothing else. Alternatively the client could supply a CompletableFuture > parameter that Solr will call complete() on etc. but that seems a bit less > natural to the notion of a method that returns it's results, albeit with a > wrapper. Proposal: > > > > > > public CompletableFuture<NamedList<Object>> > requestAsync(SolrRequest<?> request); > > > > > > BTW I'm trying to avoid implementation details here. My objective is > to devise a user-friendly API, and my hope is that the eventual > implementation is reasonable. > > > > > > I hope I haven't bored you all to tears and that you find this public > discussion useful. > > > > > > ~ David Smiley > > > Apache Lucene/Solr Search Developer > > > http://www.linkedin.com/in/davidwsmiley > > > > > > > > -- > > ----------------------------------------------------- > > Noble Paul > > > > -- > ----------------------------------------------------- > Noble Paul > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
