-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 10, 2016, 12:06 p.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available,
there is no guarantee the partition registered is complete. There can be
different ways to know if the data is complete for partition. One option could
be to have a partition property saying whether it is complete or not. Other
could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the
check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the
completeness percentage. If the completeness percentage is less than a
configured threshold (default should be 98, 99 or even 100), Lens will fail the
query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value
at query level
# Lens will still have look ahead capability of daily being incomplete, then it
will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one
with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back
with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java
fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java
4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java
408086f
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
6a0230d
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java
db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java
f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java
b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java
0aa31f4
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
cf1c233
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
PRE-CREATION
lens-server/src/main/resources/lensserver-default.xml 0ac722d
lens-server/src/main/resources/lenssession-default.xml 8d9f097
src/site/apt/admin/config.apt 21b23f9
src/site/apt/admin/session-config.apt 663a605
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
