-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36434/
-----------------------------------------------------------
(Updated July 27, 2015, 2:47 p.m.)
Review request for lens, Amareshwari Sriramadasu, Jaideep dhok, and sharad
agarwal.
Changes
-------
Pom version fixes, config-apt updates.
Bugs: LENS-252
https://issues.apache.org/jira/browse/LENS-252
Repository: lens
Description
-------
Elastic search driver for lens
~~~~~~~~~~~~~~~~~~~~~~
Elastic search accepts a nested json as a query and returns a json result.
The json result is nested for group by queries and simple for simple selects.
HQL -> ES json query
~~~~~~~~~~~~~~~~~
-> I have written a traversal (ASTTraverserForES)(specific for noSQL stores).
The traversal could be used for any purpose like query building/validation etc.
-> The traversal will take in a query visitor (ASTVisitor.java) (for building
the query) and a criteria visitor (for building the where clause). I have
checked in concrete visitors for ElasticSearch that can build the elastic
search json query.
Elasticsearch client
~~~~~~~~~~~~~~~
-> There are multiple choices of elasticsearch client available. I've made the
client pluggable.
-> I've added one default HTTPClient implementation (Jest library - apache 2).
The choice of HTTP client over transport client was made because of the version
consistency requirement between the transport client and the ES server.
-> Every client has to implement an execute method that takes in the query and
returns a LensResultSet. Hence the transformation of resultset must also be
done by the client implementation.
Elasticsearch Jest json response -> LensResultSet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The result set obtained could be a simple hit result (document response) or
facets response.
-> Facets response has a tree structure. Finding all the paths in the tree will
give us all the rows. Look at JestResultSetTransformer (AggregateTransformer)
-> Hits response is straightforward to decode (TermTransformer)
Known issues/shortcommings
~~~~~~~~~~~~~~~~~~~~~~~~
-> Scrolling responses for aggregate queries (by design ES always returns the
complete bucket in a single json - there is no scroll facility)
-> Order by in aggregate queries. Fully functional order by queries can get
complex as the ordering by measure can happen only in the immediate parent
group by. 'Limit' is also blocked as it could be misleading to have limit
without order by. (Please note that order by and limit will still work in
queries without group bys)
-> *, count is not available as of now.
-> support for other UDFs. Right now common UDAFs like sum, min, max are
supported. We need a way to seamlessly translate a new UDF to elastic search
without code change
-> Query estimation
-> Session level config injection for properties like fetch size and group by
cardinality size? (Right now these configs are at driver level)
Have added a esdriver-default.xml for looking up default properties
Diffs (updated)
-----
lens-api/src/test/java/org/apache/lens/doc/TestGenerateConfigDoc.java 5ef4186
lens-driver-es/pom.xml PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/ASTTraverserForES.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriverConfig.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESQuery.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/client/ESClient.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/client/ESResultSet.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/client/jest/JestClientImpl.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/client/jest/JestResultSetTransformer.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/exceptions/ESClientException.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/exceptions/InvalidQueryException.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ASTCriteriaVisitor.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ASTVisitor.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/CriteriaVisitorFactory.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/ESVisitor.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESAggregateVisitor.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESCriteriaVisitor.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESCriteriaVisitorFactory.java
PRE-CREATION
lens-driver-es/src/main/java/org/apache/lens/driver/es/translator/impl/ESTermVisitor.java
PRE-CREATION
lens-driver-es/src/main/resources/esdriver-default.xml PRE-CREATION
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java
PRE-CREATION
lens-driver-es/src/test/java/org/apache/lens/driver/es/MockClientES.java
PRE-CREATION
lens-driver-es/src/test/java/org/apache/lens/driver/es/QueryTranslationTest.java
PRE-CREATION
lens-driver-es/src/test/java/org/apache/lens/driver/es/ResultSetTransformationTest.java
PRE-CREATION
lens-driver-es/src/test/java/org/apache/lens/driver/es/ScrollingQueryTest.java
PRE-CREATION
lens-driver-es/src/test/resources/invalid-queries.data PRE-CREATION
lens-driver-es/src/test/resources/valid-queries.data PRE-CREATION
lens-server/pom.xml 35e7718
pom.xml 15ac998
src/site/apt/admin/config-server.apt 9349bfc
src/site/apt/admin/esdriver-config.apt PRE-CREATION
Diff: https://reviews.apache.org/r/36434/diff/
Testing
-------
Added unit test cases for testing
- query translation
- result set translation
- Scrolling
Thanks,
Amruth Sampath
