[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656419#comment-15656419
 ] 

Rajat Khandelwal commented on LENS-1359:


Taking patch from reviewboard and attaching

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch, 
> LENS-1359.12.patch, LENS-1359.13.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated LENS-1359:
---
Attachment: LENS-1359.13.patch

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch, 
> LENS-1359.12.patch, LENS-1359.13.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52924: LENS-1359: Add driver hooks for user based filtering of queries

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52924/
---

(Updated Nov. 11, 2016, 1:09 p.m.)


Review request for lens.


Bugs: LENS-1359
https://issues.apache.org/jira/browse/LENS-1359


Repository: lens


Description
---


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/parse/Parser.java PRE-CREATION 
  lens-driver-hive/src/main/resources/hivedriver-default.xml 
f5fd3bb20b9d6b82292c4483d860ea14c8104c5c 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
43b33f331f4d9017d76bccbe0f6936aaeb184f8f 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 961ec4ec781d3d7fd0f28edc4f23e9b39549592f 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
7ccb17081032024352fe03509a90ac8429254d66 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
 8f30aa06ab234cedb53acb17544210ba4190ab53 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverConfiguration.java
 69a1a0b7ce57c24602a17ea7c97c0664a77c6568 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverQueryHook.java
 f8a9ee07d8c8b737a41de339ab44ca7752c90748 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java 
146223942d05cfd952599297144e2a6bc02df273 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/NoOpDriverQueryHook.java
 4f1f2eb99ed0bec8e6e471a9723a6b002a092202 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/DriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/NoOpDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/FactPartitionBasedQueryCost.java
 476855021288ff0d91762d240cb04c8aea9d302b 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/QueryCost.java
 4712f11e94f93d34b71f4f3ddb7c19af003d9fca 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/DriverConfigurationTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/user/MockDriverQueryHook.java
 f70979abf7c44d3b4ed3588628de5a7517adfa81 
  lens-server/src/main/resources/lensserver-default.xml 
e652a0f781b0adc47b0e4bd8f11fd07eed37fa90 
  lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
50985b5447aee22b066567bf7c40fb5eee591862 
  lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml 
b485100dbe85d5b04db25ebef9affdbed8afeb31 
  pom.xml 29c59d3a06b3556ce5704bc409c4117b1ecd3f6e 
  src/site/apt/admin/config.apt f89caabeb0050df73ffc235bc3e144b64392b77d 
  src/site/apt/admin/hivedriver-config.apt 
b1a25c3b5bb58fc2d0773dc168c110ee5e60c635 

Diff: https://reviews.apache.org/r/52924/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656358#comment-15656358
 ] 

Rajat Khandelwal commented on LENS-1359:


Taking patch from reviewboard and attaching

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch, LENS-1359.12.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656364#comment-15656364
 ] 

Hadoop QA commented on LENS-1359:
-

Applied patch: 
[LENS-1359.12.patch|https://issues.apache.org/jira/secure/attachment/12838516/LENS-1359.12.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1183/

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch, LENS-1359.12.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated LENS-1359:
---
Attachment: LENS-1359.12.patch

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch, LENS-1359.12.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52924: LENS-1359: Add driver hooks for user based filtering of queries

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52924/
---

(Updated Nov. 11, 2016, 12:32 p.m.)


Review request for lens.


Bugs: LENS-1359
https://issues.apache.org/jira/browse/LENS-1359


Repository: lens


Description
---


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/parse/Parser.java PRE-CREATION 
  lens-driver-hive/src/main/resources/hivedriver-default.xml 
f5fd3bb20b9d6b82292c4483d860ea14c8104c5c 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
43b33f331f4d9017d76bccbe0f6936aaeb184f8f 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 961ec4ec781d3d7fd0f28edc4f23e9b39549592f 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
7ccb17081032024352fe03509a90ac8429254d66 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
 8f30aa06ab234cedb53acb17544210ba4190ab53 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverConfiguration.java
 69a1a0b7ce57c24602a17ea7c97c0664a77c6568 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverQueryHook.java
 f8a9ee07d8c8b737a41de339ab44ca7752c90748 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java 
146223942d05cfd952599297144e2a6bc02df273 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/NoOpDriverQueryHook.java
 4f1f2eb99ed0bec8e6e471a9723a6b002a092202 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/DriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/NoOpDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/FactPartitionBasedQueryCost.java
 476855021288ff0d91762d240cb04c8aea9d302b 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/QueryCost.java
 4712f11e94f93d34b71f4f3ddb7c19af003d9fca 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/DriverConfigurationTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/user/MockDriverQueryHook.java
 f70979abf7c44d3b4ed3588628de5a7517adfa81 
  lens-server/src/main/resources/lensserver-default.xml 
e652a0f781b0adc47b0e4bd8f11fd07eed37fa90 
  lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
50985b5447aee22b066567bf7c40fb5eee591862 
  lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml 
b485100dbe85d5b04db25ebef9affdbed8afeb31 
  pom.xml 29c59d3a06b3556ce5704bc409c4117b1ecd3f6e 
  src/site/apt/admin/config.apt f89caabeb0050df73ffc235bc3e144b64392b77d 
  src/site/apt/admin/hivedriver-config.apt 
b1a25c3b5bb58fc2d0773dc168c110ee5e60c635 

Diff: https://reviews.apache.org/r/52924/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Commented] (LENS-1333) Add data completeness checker

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656251#comment-15656251
 ] 

Hudson commented on LENS-1333:
--

UNSTABLE: Integrated in Jenkins build Lens-Commit #1375 (See 
[https://builds.apache.org/job/Lens-Commit/1375/])
LENS-1333: Add data completeness checks on query writing path (amareshwari: rev 
078555c1a92cecef19f53962703d7dd00eb44633)
* (edit) 
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
* (edit) lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java
* (edit) lens-server/src/main/resources/lenssession-default.xml
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
* (edit) lens-cube/src/main/resources/olap-query-conf.xml
* (edit) src/site/apt/user/olap-query-conf.apt
* (edit) lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java
* (edit) lens-server/src/main/resources/lensserver-default.xml
* (add) 
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
* (edit) src/site/apt/admin/session-config.apt
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
* (add) 
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java
* (delete) 
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
* (add) 
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
* (edit) src/site/apt/admin/config.apt


> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Fix For: 2.7
>
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1333) Add data completeness checker

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656209#comment-15656209
 ] 

Hudson commented on LENS-1333:
--

UNSTABLE: Integrated in Jenkins build Lens-Commit-Java8 #293 (See 
[https://builds.apache.org/job/Lens-Commit-Java8/293/])
LENS-1333: Add data completeness checks on query writing path (amareshwari: rev 
078555c1a92cecef19f53962703d7dd00eb44633)
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
* (edit) lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java
* (edit) 
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java
* (edit) lens-server/src/main/resources/lenssession-default.xml
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java
* (delete) 
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
* (edit) src/site/apt/admin/session-config.apt
* (edit) lens-server/src/main/resources/lensserver-default.xml
* (edit) lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java
* (edit) lens-cube/src/main/resources/olap-query-conf.xml
* (edit) src/site/apt/admin/config.apt
* (add) 
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
* (add) 
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
* (edit) 
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java
* (edit) src/site/apt/user/olap-query-conf.apt
* (add) 
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
* (edit) 
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java


> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Fix For: 2.7
>
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Amareshwari Sriramadasu (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656092#comment-15656092
 ] 

Amareshwari Sriramadasu commented on LENS-1359:
---

[~prongs] Patch is not applying cleanly. Can you please update to latest master?

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1372) QueryRetryTest is flaky

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated LENS-1372:
--
 Assignee: Rajat Khandelwal
Affects Version/s: 2.7
Fix Version/s: 2.7
  Component/s: test
   server

More builds failing - 
https://builds.apache.org/job/PreCommit-Lens-Build/1182/testReport/

[~prongs], please check.

> QueryRetryTest is flaky
> ---
>
> Key: LENS-1372
> URL: https://issues.apache.org/jira/browse/LENS-1372
> Project: Apache Lens
>  Issue Type: Test
>  Components: server, test
>Affects Versions: 2.7
>Reporter: Ted Yu
>Assignee: Rajat Khandelwal
>Priority: Minor
> Fix For: 2.7
>
>
> From 
> https://builds.apache.org/job/PreCommit-Lens-Build/1145/testReport/junit/org.apache.lens.server.query.retry/QueryRetryTest/testRestartWhileRetry/
>  :
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.lens.server.query.retry.QueryRetryTest.testRestartWhileRetry(QueryRetryTest.java:201)
> {code}
> Here is related code:
> {code}
> ctx = getQueryService().getQueryContext(handle);
> while (!ctx.getStatus().finished()) {
> {code}
> The test should handle the case where status is null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated LENS-1333:
--
   Resolution: Fixed
Fix Version/s: 2.7
   Status: Resolved  (was: Patch Available)

Committed. Thanks [~nperiwal]

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Fix For: 2.7
>
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1346) License mention missing for AppDispatcher.js

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654606#comment-15654606
 ] 

Hudson commented on LENS-1346:
--

UNSTABLE: Integrated in Jenkins build Lens-Commit #1374 (See 
[https://builds.apache.org/job/Lens-Commit/1374/])
LENS-1346: Mention license for AppDispatcher.js file in root LICENSE (raju: rev 
0cce226f76ad2383f904b03716ef2f5d888dc7f7)
* (edit) LICENSE


> License mention missing for AppDispatcher.js
> 
>
> Key: LENS-1346
> URL: https://issues.apache.org/jira/browse/LENS-1346
> Project: Apache Lens
>  Issue Type: Bug
>  Components: docs
>Affects Versions: 2.5
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Critical
> Fix For: 2.7
>
> Attachments: LENS-1346.2.patch, LENS-1346.patch
>
>
> Found an issue with one of the files : 
> ./lens-ui/app/dispatcher/AppDispatcher.js
> The file is having copyright from Facebook and text says the license is BSD 
> style which is in root directory of the source. But I don't see any 
> corresponding license in root directory of the source. Seems the file is 
> added in 2.5 release itself, but we need to put its corresponding license to 
> top level LICENSE file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1346) License mention missing for AppDispatcher.js

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654506#comment-15654506
 ] 

Hudson commented on LENS-1346:
--

UNSTABLE: Integrated in Jenkins build Lens-Commit-Java8 #292 (See 
[https://builds.apache.org/job/Lens-Commit-Java8/292/])
LENS-1346: Mention license for AppDispatcher.js file in root LICENSE (raju: rev 
0cce226f76ad2383f904b03716ef2f5d888dc7f7)
* (edit) LICENSE


> License mention missing for AppDispatcher.js
> 
>
> Key: LENS-1346
> URL: https://issues.apache.org/jira/browse/LENS-1346
> Project: Apache Lens
>  Issue Type: Bug
>  Components: docs
>Affects Versions: 2.5
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Critical
> Fix For: 2.7
>
> Attachments: LENS-1346.2.patch, LENS-1346.patch
>
>
> Found an issue with one of the files : 
> ./lens-ui/app/dispatcher/AppDispatcher.js
> The file is having copyright from Facebook and text says the license is BSD 
> style which is in root directory of the source. But I don't see any 
> corresponding license in root directory of the source. Seems the file is 
> added in 2.5 release itself, but we need to put its corresponding license to 
> top level LICENSE file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1346) License mention missing for AppDispatcher.js

[Raju Bairishetti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raju Bairishetti updated LENS-1346:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed patch. Thanks [~amareshwari]

> License mention missing for AppDispatcher.js
> 
>
> Key: LENS-1346
> URL: https://issues.apache.org/jira/browse/LENS-1346
> Project: Apache Lens
>  Issue Type: Bug
>  Components: docs
>Affects Versions: 2.5
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Critical
> Fix For: 2.7
>
> Attachments: LENS-1346.2.patch, LENS-1346.patch
>
>
> Found an issue with one of the files : 
> ./lens-ui/app/dispatcher/AppDispatcher.js
> The file is having copyright from Facebook and text says the license is BSD 
> style which is in root directory of the source. But I don't see any 
> corresponding license in root directory of the source. Seems the file is 
> added in 2.5 release itself, but we need to put its corresponding license to 
> top level LICENSE file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1333) Add data completeness checker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653977#comment-15653977
 ] 

Hadoop QA commented on LENS-1333:
-

Applied patch: 
[LENS-1333-v9.patch|https://issues.apache.org/jira/secure/attachment/12838354/LENS-1333-v9.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1182/

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Status: Patch Available  (was: Open)

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Attachment: LENS-1333-v9.patch

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch, LENS-1333-v9.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52514: Add data completeness checker

[Narayan Periwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
---

(Updated Nov. 10, 2016, 12:06 p.m.)


Review request for lens.


Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333


Repository: lens


Description
---

Though lens has partition registration being done whenever data is available, 
there is no guarantee the partition registered is complete. There can be 
different ways to know if the data is complete for partition. One option could 
be to have a partition property saying whether it is complete or not. Other 
could be to do a http call to another hosted service and more.

Proposal here is to add an interface for DataCompletenessChecker and do the 
check while resolving partitions.

Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the 
completeness percentage. If the completeness percentage is less than a 
configured threshold (default should be 98, 99 or even 100), Lens will fail the 
query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value 
at query level
# Lens will still have look ahead capability of daily being incomplete, then it 
will union with hourly. 
# If same measure is there in two different facts , Lens will we pick the one 
with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back 
with available percentage.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
fb958c3 
  
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 
e14c43f 
  lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
4585ef7 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
 78fb21d 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
408086f 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 
6a0230d 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 
13f1aa4 
  lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
db26718 
  lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
  lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
 7afa32e 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
f6cec1b 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
b90d4d3 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
0aa31f4 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
cf1c233 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
 PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 0ac722d 
  lens-server/src/main/resources/lenssession-default.xml 8d9f097 
  src/site/apt/admin/config.apt 21b23f9 
  src/site/apt/admin/session-config.apt 663a605 
  src/site/apt/user/olap-query-conf.apt ce35ec4 

Diff: https://reviews.apache.org/r/52514/diff/


Testing
---


Thanks,

Narayan Periwal

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Status: Open  (was: Patch Available)

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52514: Add data completeness checker

[Narayan Periwal]

> On Nov. 10, 2016, 12:02 p.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/resources/olap-query-conf.xml, line 111
> > 
> >
> > Please leave default value as blank.

Ok.


- Narayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155567
---


On Nov. 10, 2016, 12:06 p.m., Narayan Periwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> ---
> 
> (Updated Nov. 10, 2016, 12:06 p.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> 
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> 
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
> fb958c3 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
>  e14c43f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
> 4585ef7 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 
> 5dc9dc9 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
>  78fb21d 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
> 408086f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
>  6a0230d 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
>  13f1aa4 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
> db26718 
>   lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 
> 0c43cb5 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
>  7afa32e 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
>  PRE-CREATION 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
> f6cec1b 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
> b90d4d3 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
> 0aa31f4 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  cf1c233 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
>  PRE-CREATION 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
>  PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 0ac722d 
>   lens-server/src/main/resources/lenssession-default.xml 8d9f097 
>   src/site/apt/admin/config.apt 21b23f9 
>   src/site/apt/admin/session-config.apt 663a605 
>   src/site/apt/user/olap-query-conf.apt ce35ec4 
> 
> Diff: https://reviews.apache.org/r/52514/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>

Re: Review Request 52514: Add data completeness checker

[Amareshwari Sriramadasu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155567
---


Fix it, then Ship it!





lens-cube/src/main/resources/olap-query-conf.xml (line 111)


Please leave default value as blank.


- Amareshwari Sriramadasu


On Nov. 10, 2016, 10:39 a.m., Narayan Periwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> ---
> 
> (Updated Nov. 10, 2016, 10:39 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> 
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> 
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
> fb958c3 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
>  e14c43f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
> 4585ef7 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 
> 5dc9dc9 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
>  78fb21d 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
> 408086f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java
>  6a0230d 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
>  13f1aa4 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
> db26718 
>   lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 
> 0c43cb5 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
>  7afa32e 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
>  PRE-CREATION 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
> f6cec1b 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
> b90d4d3 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
> 0aa31f4 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
>  cf1c233 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
>  PRE-CREATION 
>   
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
>  PRE-CREATION 
>   lens-server/src/main/resources/lensserver-default.xml 0ac722d 
>   lens-server/src/main/resources/lenssession-default.xml 8d9f097 
>   src/site/apt/admin/config.apt 21b23f9 
>   src/site/apt/admin/session-config.apt 663a605 
>   src/site/apt/user/olap-query-conf.apt ce35ec4 
> 
> Diff: https://reviews.apache.org/r/52514/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>

[jira] [Commented] (LENS-1333) Add data completeness checker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653799#comment-15653799
 ] 

Hadoop QA commented on LENS-1333:
-

Applied patch: 
[LENS-1333-v8.patch|https://issues.apache.org/jira/secure/attachment/12838338/LENS-1333-v8.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1181/

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Status: Patch Available  (was: Open)

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52514: Add data completeness checker

[Narayan Periwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
---

(Updated Nov. 10, 2016, 10:39 a.m.)


Review request for lens.


Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333


Repository: lens


Description
---

Though lens has partition registration being done whenever data is available, 
there is no guarantee the partition registered is complete. There can be 
different ways to know if the data is complete for partition. One option could 
be to have a partition property saying whether it is complete or not. Other 
could be to do a http call to another hosted service and more.

Proposal here is to add an interface for DataCompletenessChecker and do the 
check while resolving partitions.

Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the 
completeness percentage. If the completeness percentage is less than a 
configured threshold (default should be 98, 99 or even 100), Lens will fail the 
query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value 
at query level
# Lens will still have look ahead capability of daily being incomplete, then it 
will union with hourly. 
# If same measure is there in two different facts , Lens will we pick the one 
with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back 
with available percentage.


Diffs (updated)
-

  lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
fb958c3 
  
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 
e14c43f 
  lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
4585ef7 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
 78fb21d 
  lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
408086f 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 
6a0230d 
  
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 
13f1aa4 
  lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
db26718 
  lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
  lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
 7afa32e 
  
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java 
PRE-CREATION 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
f6cec1b 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
b90d4d3 
  lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
0aa31f4 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
cf1c233 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java
 PRE-CREATION 
  lens-server/src/main/resources/lensserver-default.xml 0ac722d 
  lens-server/src/main/resources/lenssession-default.xml 8d9f097 
  src/site/apt/admin/config.apt 21b23f9 
  src/site/apt/admin/session-config.apt 663a605 
  src/site/apt/user/olap-query-conf.apt ce35ec4 

Diff: https://reviews.apache.org/r/52514/diff/


Testing
---


Thanks,

Narayan Periwal

[jira] [Commented] (LENS-1371) Add a substring based retry policy for failures

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653711#comment-15653711
 ] 

Hadoop QA commented on LENS-1371:
-

Applied patch: 
[0001-Retry-policy.patch|https://issues.apache.org/jira/secure/attachment/12838142/0001-Retry-policy.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1180/

> Add a substring based retry policy for failures
> ---
>
> Key: LENS-1371
> URL: https://issues.apache.org/jira/browse/LENS-1371
> Project: Apache Lens
>  Issue Type: New Feature
>Reporter: Lavkesh Lahngir
>Assignee: Lavkesh Lahngir
> Attachments: 0001-Retry-patch.patch, 0001-Retry-policy.patch, 
> 0001-Retry-policy.patch, 0001-Retry-policy.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Attachment: LENS-1333-v8.patch

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch, LENS-1333-v8.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1333) Add data completeness checker

[Narayan Periwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narayan Periwal updated LENS-1333:
--
Status: Open  (was: Patch Available)

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52514: Add data completeness checker

[Narayan Periwal]

> On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java,
> >  line 99
> > 
> >
> > call it isDataCompletenessCheckEnabled or dataCompletenessChecksEnabled 
> > ? 
> > 
> > is and checks in the does not look good.

Sure..Will correct it.


> On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, 
> > line 49
> > 
> >
> > Add entry in olap-query-conf.xml with description.

Agreed.


- Narayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review14
---


On Nov. 10, 2016, 7:09 a.m., Narayan Periwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> ---
> 
> (Updated Nov. 10, 2016, 7:09 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> 
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> 
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.
> 
> 
> Diffs
> -
> 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java 
> fb958c3 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
>  e14c43f 
>   
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 
> 4585ef7 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 
> 5dc9dc9 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java
>  78fb21d 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 
> 408086f 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java 
> PRE-CREATION 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java
>  13f1aa4 
>   
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java 
> db26718 
>   lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 
> 0c43cb5 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java
>  7afa32e 
>   
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java
>  PRE-CREATION 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 
> f6cec1b 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java 
> b90d4d3 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 
> 0aa31f4 
>   lens-server/src/main/resources/lenssession-default.xml 8d9f097 
>   src/site/apt/admin/session-config.apt 663a605 
>   src/site/apt/user/olap-query-conf.apt ce35ec4 
> 
> Diff: https://reviews.apache.org/r/52514/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653596#comment-15653596
 ] 

Hadoop QA commented on LENS-1359:
-

Applied patch: 
[LENS-1359.11.patch|https://issues.apache.org/jira/secure/attachment/12838320/LENS-1359.11.patch]
 and ran command: mvn clean install -fae. Result: Success. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1179/

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653483#comment-15653483
 ] 

Hadoop QA commented on LENS-1359:
-

Applied patch: 
[LENS-1359.11.patch|https://issues.apache.org/jira/secure/attachment/12838320/LENS-1359.11.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1178/

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated LENS-1359:
---
Attachment: LENS-1359.11.patch

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653392#comment-15653392
 ] 

Rajat Khandelwal commented on LENS-1359:


Taking patch from reviewboard and attaching

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch, LENS-1359.11.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on LENS-1359 started by Rajat Khandelwal.
--
> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (LENS-1359) Add driver hooks for user based filtering of queries

[Rajat Khandelwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LENS-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajat Khandelwal updated LENS-1359:
---
Status: Patch Available  (was: In Progress)

> Add driver hooks for user based filtering of queries
> 
>
> Key: LENS-1359
> URL: https://issues.apache.org/jira/browse/LENS-1359
> Project: Apache Lens
>  Issue Type: Task
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
>  Labels: incompatible
> Attachments: LENS-1359.01.patch, LENS-1359.02.patch, 
> LENS-1359.03.patch, LENS-1359.04.patch, LENS-1359.05.patch, 
> LENS-1359.06.patch, LENS-1359.07.patch, LENS-1359.08.patch, 
> LENS-1359.09.patch, LENS-1359.10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52924: LENS-1359: Add driver hooks for user based filtering of queries

[Rajat Khandelwal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52924/
---

(Updated Nov. 10, 2016, 1:41 p.m.)


Review request for lens.


Bugs: LENS-1359
https://issues.apache.org/jira/browse/LENS-1359


Repository: lens


Description
---


Diffs (updated)
-

  lens-api/src/main/java/org/apache/lens/api/parse/Parser.java PRE-CREATION 
  lens-driver-hive/src/main/resources/hivedriver-default.xml 
f5fd3bb20b9d6b82292c4483d860ea14c8104c5c 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 
43b33f331f4d9017d76bccbe0f6936aaeb184f8f 
  
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java
 961ec4ec781d3d7fd0f28edc4f23e9b39549592f 
  
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 
cf1c233a58fd8b5f5ead95b3fe9066d63976d595 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java
 8f30aa06ab234cedb53acb17544210ba4190ab53 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverConfiguration.java
 69a1a0b7ce57c24602a17ea7c97c0664a77c6568 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverQueryHook.java
 f8a9ee07d8c8b737a41de339ab44ca7752c90748 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java 
146223942d05cfd952599297144e2a6bc02df273 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/NoOpDriverQueryHook.java
 4f1f2eb99ed0bec8e6e471a9723a6b002a092202 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/DriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/NoOpDriverQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHook.java
 PRE-CREATION 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/FactPartitionBasedQueryCost.java
 476855021288ff0d91762d240cb04c8aea9d302b 
  
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/QueryCost.java
 4712f11e94f93d34b71f4f3ddb7c19af003d9fca 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/DriverConfigurationTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/ChainedDriverQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/QueryCostBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/driver/hooks/UserBasedQueryHookTest.java
 PRE-CREATION 
  
lens-server-api/src/test/java/org/apache/lens/server/api/user/MockDriverQueryHook.java
 f70979abf7c44d3b4ed3588628de5a7517adfa81 
  lens-server/src/main/resources/lensserver-default.xml 
0ac722d7788f610d2e4b81cef8f1f120dcf221ca 
  lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml 
50985b5447aee22b066567bf7c40fb5eee591862 
  lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml 
b485100dbe85d5b04db25ebef9affdbed8afeb31 
  pom.xml 29c59d3a06b3556ce5704bc409c4117b1ecd3f6e 
  src/site/apt/admin/config.apt 21b23f9075326b4a10ab57eb1febf06f81f33b4b 
  src/site/apt/admin/hivedriver-config.apt 
b1a25c3b5bb58fc2d0773dc168c110ee5e60c635 

Diff: https://reviews.apache.org/r/52924/diff/


Testing
---


Thanks,

Rajat Khandelwal

[jira] [Commented] (LENS-1333) Add data completeness checker

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LENS-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653362#comment-15653362
 ] 

Hadoop QA commented on LENS-1333:
-

Applied patch: 
[LENS-1333-v7.patch|https://issues.apache.org/jira/secure/attachment/12838306/LENS-1333-v7.patch]
 and ran command: mvn clean install -fae. Result: Failure. Build Job: 
https://builds.apache.org/job/PreCommit-Lens-Build/1177/

> Add data completeness checker
> -
>
> Key: LENS-1333
> URL: https://issues.apache.org/jira/browse/LENS-1333
> Project: Apache Lens
>  Issue Type: New Feature
>  Components: cube
>Reporter: Amareshwari Sriramadasu
>Assignee: Narayan Periwal
> Attachments: LENS-1333-v1.patch, LENS-1333-v2.patch, 
> LENS-1333-v3.patch, LENS-1333-v4.patch, LENS-1333-v5.patch, 
> LENS-1333-v6.patch, LENS-1333-v7.patch
>
>
> Though lens has partition registration being done whenever data is available, 
> there is no guarantee the partition registered is complete. There can be 
> different ways to know if the data is complete for partition. One option 
> could be to have a partition property saying whether it is complete or not. 
> Other could be to do a http call to another hosted service and more.
> Proposal here is to add an interface for DataCompletenessChecker and do the 
> check while resolving partitions.
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the 
> completeness percentage. If the completeness percentage is less than a 
> configured threshold (default should be 98, 99 or even 100), Lens will fail 
> the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold 
> value at query level
> # Lens will still have look ahead capability of daily being incomplete, then 
> it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one 
> with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back 
> with available percentage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)