Ok. Submitting the patch. On Wed, Jul 29, 2015 at 11:37 AM, Amareshwari Sriramadasu < [email protected]> wrote:
> > > > On July 28, 2015, 5:27 a.m., Amareshwari Sriramadasu wrote: > > > Please update summary from mvn clean install > > > > Amruth Sampath wrote: > > Hi I have updated the revision again which has a critical fix for > the cube queries to work. > > This is for figuring out the underlying schema (index) and table > (type) and rewriting the hql (using the properties > lens.metastore.native.db.name and lens.metastore.native.table.name). > > > > Kindly take a look at that also and let me know if its good to go. > > > > Amruth Sampath wrote: > > Have also added support for rewriting between clause in elastic > searc (missed in the previous comment) > > I'm thinking the driver can go in. And we need to pick more tasks on top > of it to integrate with cube queries with end to end testcases, adding > integration tests to run queries along with other drivers (testing cost > computation and comparison), add examples with tables and data, > illustrating this driver. > > > - Amareshwari > > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36434/#review93238 > ----------------------------------------------------------- > > > On July 28, 2015, 8:53 p.m., Amruth Sampath wrote: > > > > ----------------------------------------------------------- > > This is an automatically generated e-mail. To reply, visit: > > https://reviews.apache.org/r/36434/ > > ----------------------------------------------------------- > > > > (Updated July 28, 2015, 8:53 p.m.) > > > > > > Review request for lens, Amareshwari Sriramadasu, Jaideep dhok, and > sharad agarwal. > > > > > > Bugs: LENS-252 > > https://issues.apache.org/jira/browse/LENS-252 > > > > > > Repository: lens > > > > > > Description > > ------- > > > > Elastic search driver for lens > > ~~~~~~~~~~~~~~~~~~~~~~ > > Elastic search accepts a nested json as a query and returns a json > result. > > The json result is nested for group by queries and simple for simple > selects. > > HQL -> ES json query > > ~~~~~~~~~~~~~~~~~ > > -> I have written a traversal (ASTTraverserForES)(specific for noSQL > stores). The traversal could be used for any purpose like query > building/validation etc. > > -> The traversal will take in a query visitor (ASTVisitor.java) (for > building the query) and a criteria visitor (for building the where clause). > I have checked in concrete visitors for ElasticSearch that can build the > elastic search json query. > > Elasticsearch client > > ~~~~~~~~~~~~~~~ > > -> There are multiple choices of elasticsearch client available. I've > made the client pluggable. > > -> I've added one default HTTPClient implementation (Jest library - > apache 2). The choice of HTTP client over transport client was made because > of the version consistency requirement between the transport client and the > ES server. > > -> Every client has to implement an execute method that takes in the > query and returns a LensResultSet. Hence the transformation of resultset > must also be done by the client implementation. > > Elasticsearch Jest json response -> LensResultSet > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > The result set obtained could be a simple hit result (document response) > or facets response. > > -> Facets response has a tree structure. Finding all the paths in the > tree will give us all the rows. Look at JestResultSetTransformer > (AggregateTransformer) > > -> Hits response is straightforward to decode (TermTransformer) > > Known issues/shortcommings > > ~~~~~~~~~~~~~~~~~~~~~~~~ > > -> Scrolling responses for aggregate queries (by design ES always > returns the complete bucket in a single json - there is no scroll facility) > > -> Order by in aggregate queries. Fully functional order by queries can > get complex as the ordering by measure can happen only in the immediate > parent group by. 'Limit' is also blocked as it could be misleading to have > limit without order by. (Please note that order by and limit will still > work in queries without group bys) > > -> *, count is not available as of now. > > -> support for other UDFs. Right now common UDAFs like sum, min, max are > supported. We need a way to seamlessly translate a new UDF to elastic > search without code change > > -> Query estimation > > -> Session level config injection for properties like fetch size and > group by cardinality size? (Right now these configs are at driver level) > > Have added a esdriver-default.xml for looking up default properties > > > > > > Diffs > > ----- > > > > lens-api/src/test/java/org/apache/lens/doc/TestGenerateConfigDoc.java > 5ef4186 > > lens-driver-es/pom.xml PRE-CREATION > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/ASTTraverserForES.java > PRE-CREATION > > lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java > PRE-CREATION > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriverConfig.java > PRE-CREATION > > lens-driver-es/src/main/java/org/apache/lens/driver/es/ESQuery.java > PRE-CREATION > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/client/ESClient.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/client/ESResultSet.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/client/jest/JestClientImpl.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/client/jest/JestResultSetTransformer.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/exceptions/ESClientException.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/exceptions/InvalidQueryException.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ASTCriteriaVisitor.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ASTVisitor.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/CriteriaVisitorFactory.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ESVisitor.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESAggregateVisitor.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESCriteriaVisitor.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESCriteriaVisitorFactory.java > PRE-CREATION > > > > lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESTermVisitor.java > PRE-CREATION > > lens-driver-es/src/main/resources/esdriver-default.xml PRE-CREATION > > > lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java > PRE-CREATION > > > lens-driver-es/src/test/java/org/apache/lens/driver/es/MockClientES.java > PRE-CREATION > > > > lens-driver-es/src/test/java/org/apache/lens/driver/es/QueryTranslationTest.java > PRE-CREATION > > > > lens-driver-es/src/test/java/org/apache/lens/driver/es/ResultSetTransformationTest.java > PRE-CREATION > > > > lens-driver-es/src/test/java/org/apache/lens/driver/es/ScrollingQueryTest.java > PRE-CREATION > > lens-driver-es/src/test/resources/invalid-queries.data PRE-CREATION > > lens-driver-es/src/test/resources/valid-queries.data PRE-CREATION > > lens-server/pom.xml 35e7718 > > pom.xml 15ac998 > > src/site/apt/admin/config-server.apt 9349bfc > > src/site/apt/admin/esdriver-config.apt PRE-CREATION > > src/site/apt/user/cli.apt a28d8a4 > > > > Diff: https://reviews.apache.org/r/36434/diff/ > > > > > > Testing > > ------- > > > > Added unit test cases for testing > > - query translation > > - result set translation > > - Scrolling > > > > > > Thanks, > > > > Amruth Sampath > > > > > > -- Amruth S (09486075517) -- ------------------------------------------------------------------------------------------------------------------------------------------ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Although Flipkart has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments
