Hi Kasper, Your idea is great I will make a "first functional version" and then we might improve it with native ES queries and filters.
I haven´t made an ICLA for ASF yet. I will send the signed form ASAP. I am also going to fork the MM project, do you want me to create a new branch from master on the forked project and start making my changes there? Thank you! Kind regards, Alberto 2014-09-12 22:39 GMT+02:00 Kasper Sørensen <[email protected]>: > Hi Alberto, > > We agree completely :-) it's great that you push as much querying down to > the backend. I was only suggesting you start with making something that is > "slow and stupid, but functional" and then optimize. But I think you're > already doing much better :-) > > Consider making this as a fork to the MetaModel source. At least, I think > we could integrate this quite soon into the main MM source tree. > > I do think however we would need a contributor license agreement from you > to accept it (it's not just a patch, but more substantial). Have you > already made an ICLA for ASF? > > Best regards, > Kasper > > 2014-09-12 22:10 GMT+02:00 Alberto Rodriguez <[email protected]>: > > > Hi Kasper, > > > > Ok got it! thank you for explaining to me. > > > > I will implement the materializeMainSchemaTable but as you said I think > is > > a good idea to take advantage of the very powerful ElasticSearch API > which > > has a DSL that support natively queries and filters, have a look at them: > > Query > > DSL > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/query-dsl-queries.html > > > > > , Query dsl filters > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/query-dsl-filters.html > > >. > > By using these APIs we would also get a huge performance improvement > > because we were using Apache Lucene under-the-hood. > > > > I will let you know if there is any further update. Thank you!! > > > > Kind regards, > > > > Alberto > > > > > > > > > > 2014-09-12 14:52 GMT+02:00 Kasper Sørensen < > [email protected] > > >: > > > > > Hi Alberto, > > > > > > Yes you are right that these ones do override executeQuery(Query). In > the > > > case of JDBC, it's a completely different story so let's leave that out > > :-) > > > In the case of MongoDB this method is overridden to catch a few > optimized > > > corner cases, which is fine. But in case those corner cases are not > met, > > it > > > delegates to super.executeQuery(Query), which in turn will delegate to > > > materializeMainSchemaTable(...) then! That is why, if you implement > > > materializeMainSchemaTable(...) then at least you are 100% functional, > > > because MM can then always revert to doing any filtering and > aggregation > > in > > > memory! It may not be performing well at all, but it's a great start to > > be > > > functional and compatible with the query model. > > > > > > I would in deed recommend you catch the corner cases of the WHERE items > > > that you can support natively. And also to implement > > executeCountQuery(...) > > > would be good. > > > > > > Best regards, > > > Kasper > > > > > > 2014-09-12 14:35 GMT+02:00 Alberto Rodriguez <[email protected]>: > > > > > > > Hi Kasper, > > > > > > > > thank you for the feedback! > > > > > > > > I am not sure if I fully understand your piece of advice related to > not > > > > implement the executeQuery(query) method. I've been having a look at > > the > > > > different modules (jdbc, mongo...) and they do override this method. > > I've > > > > just taken a look at the materializeTable method signature and it > does > > > not > > > > include, for instance, the FilterItem list so I wouldn't have acces > to > > > the > > > > filters within this method. > > > > > > > > I've just commited some changes that allow to add simple > where/isEquals > > > > clause to the query (although they are inside the executeQuery). > > > > > > > > I have also just upgraded the dependency to the latest version (thank > > > you!) > > > > > > > > Kind regards, > > > > > > > > Alberto > > > > > > > > 2014-09-12 13:58 GMT+02:00 Kasper Sørensen < > > > [email protected] > > > > >: > > > > > > > > > Hi Alberto, > > > > > > > > > > I gave you project a quick look, and it's already quite awesome > work! > > > > :-) I > > > > > like the way you discover metadata ... I didn't know of that > > approach, > > > > so I > > > > > am looking forward to trying it out. > > > > > > > > > > A quick thing I noticed is that your DataContext overrides the > > > > > executeQuery(Query) method. You probably shouldn't do that, since > > your > > > > > superclass does all the hard work. Simply implement the > > > > > materializeTable(...) method and you're much better off. > > > > > > > > > > If you upgrade your dependency to rely on the latest version of MM > > > (4.2) > > > > > then you will even have a lot more helper methods to implement in > > your > > > > > DataContext. Such as executePrimaryKeyLookupQuery and others. Doing > > > that > > > > > will quickly get your going to a very broad query support and > better > > > > > performance. > > > > > > > > > > Best regards, > > > > > Kasper > > > > > > > > > > > > > > > 2014-09-12 13:27 GMT+02:00 Hans Drexler < > > > [email protected] > > > > >: > > > > > > > > > > > Just a small correction. MetaModel is not my project. I am not > > nearly > > > > > > smart enough for that. But I'll do my best to give you some > > > worthwhile > > > > > > feedback. > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Hans > > > > > > > > > > > > -----Original Message----- > > > > > > From: Alberto Rodriguez [mailto:[email protected]] > > > > > > Sent: Friday, September 12, 2014 1:03 PM > > > > > > To: [email protected] > > > > > > Subject: Re: ElasticSearch Module > > > > > > > > > > > > Hi Hans, > > > > > > > > > > > > no worries!! I think your project is awesome and I would like to > > > > > > contribute somehow. > > > > > > > > > > > > I will keep working on the module. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Alberto > > > > > > > > > > > > 2014-09-12 12:58 GMT+02:00 Hans Drexler < > > > > [email protected] > > > > > >: > > > > > > > > > > > > > Hi Alberto, > > > > > > > > > > > > > > What great news. I will checkout your code and try to give > > > comments. > > > > > > > Thank you for your contributon. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Hans > > > > > > > > > > > > > > Op 12 sep. 2014 11:13 schreef Alberto Rodriguez < > > [email protected] > > > >: > > > > > > > Hi there, > > > > > > > > > > > > > > as per our previous discussion I have been taking a look at the > > > > > > > elasticsearch module for metamodel. > > > > > > > > > > > > > > I have been working on it and I have a very simple version > which > > is > > > > > > > already fetching the metadata from ES. I have also managed to > > make > > > > > > > simple queries work (queries without filters and where > clauses). > > > > > > > > > > > > > > Could you please have a look at the project on my github > account > > > and > > > > > > > let me know what you think about it?: Metamodel-elasticsearch > > > module > > > > > > > <https://github.com/ardlema/metamodel-elasticsearch> > > > > > > > > > > > > > > Thank you in advance. > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > > > Alberto > > > > > > > > > > > > > > > > > > > > > > > > > > > >
