[
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656209#comment-15656209
]
Hudson commented on LENS-1333:
------------------------------
UNSTABLE: Integrated in Jenkins build Lens-Commit-Java8 #293 (See
[https://builds.apache.org/job/Lens-Commit-Java8/293/])
LENS-1333: Add data completeness checks on query writing path (amareshwari: rev
078555c1a92cecef19f53962703d7dd00eb44633)
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
* (edit) lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
* (edit)
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java
* (edit)
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java
* (edit) lens-server/src/main/resources/lenssession-default.xml
* (edit)
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java
* (delete)
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
* (edit) src/site/apt/admin/session-config.apt
* (edit) lens-server/src/main/resources/lensserver-default.xml
* (edit) lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java
* (edit) lens-cube/src/main/resources/olap-query-conf.xml
* (edit) src/site/apt/admin/config.apt
* (add)
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
* (add)
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
* (edit)
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
* (edit)
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java
* (edit) src/site/apt/user/olap-query-conf.apt
* (add)
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
* (edit)
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
> Add data completeness checker
> -----------------------------
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
> Issue Type: New Feature
> Components: cube
> Reporter: Amareshwari Sriramadasu
> Assignee: Narayan Periwal
> Fix For: 2.7
>
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch,
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch,
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available,
> there is no guarantee the partition registered is complete. There can be
> different ways to know if the data is complete for partition. One option
> could be to have a partition property saying whether it is complete or not.
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the
> completeness percentage. If the completeness percentage is less than a
> configured threshold (default should be 98, 99 or even 100), Lens will fail
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then
> it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back
> with available percentage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
