> On Nov. 10, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > Overall changes look fine. But I'm just wondering it would be good if we 
> > can have a configuration to enable/disable the feature fully - 
> > lens.cube.metastore.enable.datacompleteness.checks - if it is set to false, 
> > it can fall back to the old code, and not otherwise.

Agreed. It makes sense to add this to enable/disable the feature from this 
config.


- Narayan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155550
-----------------------------------------------------------


On Nov. 9, 2016, 1:39 p.m., Narayan Periwal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
> 
> (Updated Nov. 9, 2016, 1:39 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1333
>     https://issues.apache.org/jira/browse/LENS-1333
> 
> 
> Repository: lens
> 
> 
> Description
> -------
> 
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> 
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> 
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.
> 
> 
> Diffs
> -----
> 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
> fb958c3 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
>  e14c43f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
> 4585ef7 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 
> 5dc9dc9 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
>  78fb21d 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
> f20f105 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java 
> PRE-CREATION 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
>  13f1aa4 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
> db26718 
>   lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 
> 0c43cb5 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
>  7afa32e 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
>  PRE-CREATION 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
> f6cec1b 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
> b90d4d3 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
> 0aa31f4 
>   lens-server/src/main/resources/lenssession-default.xml 9a8f9da 
>   src/site/apt/admin/session-config.apt d480f88 
>   src/site/apt/user/olap-query-conf.apt ce35ec4 
> 
> Diff: https://reviews.apache.org/r/52514/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>

Reply via email to