-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------

(Updated Nov. 10, 2016, 12:06 p.m.)


Review request for lens.


Bugs: LENS-1333
    https://issues.apache.org/jira/browse/LENS-1333


Repository: lens


Description
-------

Though lens has partition registration being done whenever data is available, 
there is no guarantee the partition registered is complete. There can be 
different ways to know if the data is complete for partition. One option could 
be to have a partition property saying whether it is complete or not. Other 
could be to do a http call to another hosted service and more.

Proposal here is to add an interface for DataCompletenessChecker and do the 
check while resolving partitions.

Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the 
completeness percentage. If the completeness percentage is less than a 
configured threshold (default should be 98, 99 or even 100), Lens will fail the 
query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value 
at query level
# Lens will still have look ahead capability of daily being incomplete, then it 
will union with hourly. 
# If same measure is there in two different facts , Lens will we pick the one 
with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back 
with available percentage.


Diffs (updated)
-----

  lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
fb958c3 
  
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 
e14c43f 
  lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
4585ef7 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
 78fb21d 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
408086f 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 
6a0230d 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 
13f1aa4 
  lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
db26718 
  lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
  lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
 7afa32e 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
f6cec1b 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
b90d4d3 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
0aa31f4 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
cf1c233 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
 PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 0ac722d 
  lens-server/src/main/resources/lenssession-default.xml 8d9f097 
  src/site/apt/admin/config.apt 21b23f9 
  src/site/apt/admin/session-config.apt 663a605 
  src/site/apt/user/olap-query-conf.apt ce35ec4 

Diff: https://reviews.apache.org/r/52514/diff/


Testing
-------


Thanks,

Narayan Periwal

Reply via email to