Github user kaspersorensen commented on the pull request:
https://github.com/apache/metamodel/pull/19#issuecomment-98796448
Hi Ashish,
I know that doing a JOIN is pretty unusual in "normal" mode of Solr. But in
my opinion we are also not catering to the normal cases. MetaModel exposes an
API that is useful especially because it is the _same_ API across all
datastores. That means our users often use MetaModel in scenarios where they
might just as well use a relational database or an Excel sheet as the backing
store. The benefit they get with MetaModel is that their code to process data
can run on any given datastore. That means the type of use is often pretty
untypical. So having support for JOIN in Solr is actually pretty important in
my opinion. I would even go as far as to say that effeciency in execution is
secondary to compatibility with the API. Reason being that if people are trying
to get the most performance out of their Solr system then they will anyways be
going directly to Solr's primary API.
That said I do believe that there are scenarios you absolutely should
support.
My suggestion is just that you start small and get something that
functionally fulfills the whole API. Then expand on a case-by-case basis to add
optimized support for various classes of queries. For instance you can start
with just materializeMainSchemaTable - make it work without any default limits
or anything like that. And make sure that it is cursor based so that you don't
load a million records into memory but rather maintain a cursor in the dataset
implementation.
Once that is working I would expand the scope to also have support for
COUNT queries. And then maybe queries with just a single WHERE item. And so on
and so on ... Makes sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
