[ https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557107#comment-15557107 ]
Narayan Periwal commented on LENS-1333: --------------------------------------- [~prongs], actually it's a compilation error {code} [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-Lens-Build/lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java:[124,39] cannot find symbol symbol: class DataCompletenessChecker location: class org.apache.lens.cube.parse.CubeQueryConfUtil [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-Lens-Build/lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java:[71,11] cannot find symbol symbol: class DataCompletenessChecker location: class org.apache.lens.cube.parse.StorageTableResolver [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-Lens-Build/lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java:[27,40] cannot find symbol symbol: class DataCompletenessChecker {code} This will get compiled once LENS-1340 gets checked in. As the changes for LENS-1340 got reverted, therefore it's not compiling. > Add data completeness checker > ----------------------------- > > Key: LENS-1333 > URL: https://issues.apache.org/jira/browse/LENS-1333 > Project: Apache Lens > Issue Type: New Feature > Components: cube > Reporter: Amareshwari Sriramadasu > Assignee: Narayan Periwal > Attachments: LENS-1333-v1.patch > > > Though lens has partition registration being done whenever data is available, > there is no guarantee the partition registered is complete. There can be > different ways to know if the data is complete for partition. One option > could be to have a partition property saying whether it is complete or not. > Other could be to do a http call to another hosted service and more. > Proposal here is to add an interface for DataCompletenessChecker and do the > check while resolving partitions. > Here are some of the capabilities we would like to add in Lens : > # Lens will check partition existence first, if it exists, then check the > completeness percentage. If the completeness percentage is less than a > configured threshold (default should be 98, 99 or even 100), Lens will fail > the query. > # Lens's accept query on partial data will accept on incomplete data as well. > # Lens will also option to override the completeness percentage threshold > value at query level > # Lens will still have look ahead capability of daily being incomplete, then > it will union with hourly. > # If same measure is there in two different facts , Lens will we pick the one > with higher availability. > # In case of completeness percentage threshold missed, Lens will respond back > with available percentage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)