-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36884/#review93273
-----------------------------------------------------------
Here is the log estimate on example query :
28 Jul 2015 16:47:37,797 [b6b7a2bf-c4eb-418c-8377-7e73c86621b2]
[Grizzly-worker(8)] INFO
org.apache.lens.server.query.QueryExecutionServiceImpl - ExecuteAsync:
session:<?xml version="1.0" encoding="UTF-8"
standalone="yes"?><lensSessionHandle><publicId>5dba21c3-2a0e-4d51-9e55-a22e29f854b8</publicId><secretId>120a77c2-483a-429d-933c-ef3fead506bb</secretId></lensSessionHandle>
query: cube select measure2 from sample_cube where time_range_in(dt,
'2014-06-24-23', '2014-06-25-00')
28 Jul 2015 16:47:37,799 [b6b7a2bf-c4eb-418c-8377-7e73c86621b2]
[Grizzly-worker(8)] INFO
org.apache.lens.server.query.QueryExecutionServiceImpl - Returning handle
53deaaa2-31a6-4f0e-a46f-47f2f00af15e
28 Jul 2015 16:47:37,799 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e]
[QuerySubmitter] INFO org.apache.lens.server.query.QueryExecutionServiceImpl
- Processing query:cube select measure2 from sample_cube where
time_range_in(dt, '2014-06-24-23', '2014-06-25-00')
28 Jul 2015 16:47:37,829 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e]
[QuerySubmitter] INFO hive.ql.parse.ParseDriver - Parsing command: cube
select measure2 from sample_cube where time_range_in(dt, '2014-06-24-23',
'2014-06-25-00')
28 Jul 2015 16:47:37,831 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e]
[QuerySubmitter] INFO hive.ql.parse.ParseDriver - Parse Completed
28 Jul 2015 16:47:37,833 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO hive.ql.parse.ParseDriver - Parsing command: cube select measure2 from
sample_cube where time_range_in(dt, '2014-06-24-23', '2014-06-25-00')
28 Jul 2015 16:47:37,833 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO hive.ql.parse.ParseDriver - Parsing command: cube select measure2 from
sample_cube where time_range_in(dt, '2014-06-24-23', '2014-06-25-00')
28 Jul 2015 16:47:37,835 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO hive.ql.parse.ParseDriver - Parse Completed
28 Jul 2015 16:47:37,835 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO hive.ql.parse.ParseDriver - Parse Completed
28 Jul 2015 16:47:37,836 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CandidateTableResolver - Populated candidate
facts: [fact2, rawfact, fact1]
28 Jul 2015 16:47:37,836 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CandidateTableResolver - Measure covering
fact sets :[[fact2], [rawfact], [fact1]]
28 Jul 2015 16:47:37,837 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.JoinResolver - No dimension tables to resolve
and no join chains present!
28 Jul 2015 16:47:37,836 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.CandidateTableResolver - Populated candidate
facts: [fact2, rawfact, fact1]
28 Jul 2015 16:47:37,837 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.CandidateTableResolver - Measure covering
fact sets :[[fact2], [rawfact], [fact1]]
28 Jul 2015 16:47:37,838 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.JoinResolver - No dimension tables to resolve
and no join chains present!
28 Jul 2015 16:47:37,839 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
fact2 from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Wed Jun 25 00:00:00 IST
2014
28 Jul 2015 16:47:37,839 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
fact2 from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Tue Jun 24 23:00:00 IST
2014
28 Jul 2015 16:47:37,839 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Skipping storage:
local as it is not supported
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
fact2 from fromDate:Wed Jun 25 00:00:00 IST 2014 toDate:Wed Jun 25 00:00:00 IST
2014
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Not considering fact
table:fact2 as it does not have any storage tables
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Skipping storage:
local as it is not supported
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Not considering fact
table:rawfact as it does not have any storage tables
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Skipping storage:
local as it is not supported
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - Resolved partitions
for fact fact2: [dt = '2014-06-24-23'] storageTables:[local_fact2]
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.StorageTableResolver - Not considering fact
table:fact1 as it does not have any storage tables
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
rawfact from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Wed Jun 25 00:00:00
IST 2014
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.CubeQueryContext - Not considering fact table
set:[fact2] as they have non candidate tables and facts missing because of
NO_CANDIDATE_STORAGES
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.CubeQueryContext - Not considering fact table
set:[rawfact] as they have non candidate tables and facts missing because of
NO_CANDIDATE_STORAGES
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
INFO org.apache.lens.cube.parse.CubeQueryContext - Not considering fact table
set:[fact1] as they have non candidate tables and facts missing because of
NO_CANDIDATE_STORAGES
28 Jul 2015 16:47:37,840 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - Partition:dt =
'2014-06-24-23' does not exist in any storage table
28 Jul 2015 16:47:37,841 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
rawfact from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Wed Jun 25 00:00:00
IST 2014
28 Jul 2015 16:47:37,841 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - No max interval for
range: Tue Jun 24 23:00:00 IST 2014 to Wed Jun 25 00:00:00 IST 2014
28 Jul 2015 16:47:37,842 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
rawfact from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Tue Jun 24 23:00:00
IST 2014
28 Jul 2015 16:47:37,842 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
rawfact from fromDate:Wed Jun 25 00:00:00 IST 2014 toDate:Wed Jun 25 00:00:00
IST 2014
28 Jul 2015 16:47:37,843 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - No partitions for
range:dt [2014-06-24-23:00:00,000 to 2014-06-25-00:00:00,000). fallback range:
null
28 Jul 2015 16:47:37,844 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - No partitions for
fallback range:dt [2014-06-24-23:00:00,000 to 2014-06-25-00:00:00,000)
28 Jul 2015 16:47:37,846 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - Not considering fact
table:rawfact as it could not find partition for given ranges: [dt
[2014-06-24-23:00:00,000 to 2014-06-25-00:00:00,000)]
28 Jul 2015 16:47:37,846 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
fact1 from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Wed Jun 25 00:00:00 IST
201428 Jul 2015 16:47:37,846 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e]
[estimate-3] INFO org.apache.lens.cube.parse.StorageTableResolver -
getPartitions for fact1 from fromDate:Tue Jun 24 23:00:00 IST 2014 toDate:Tue
Jun 24 23:00:00 IST 2014
28 Jul 2015 16:47:37,847 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - getPartitions for
fact1 from fromDate:Wed Jun 25 00:00:00 IST 2014 toDate:Wed Jun 25 00:00:00 IST
2014
28 Jul 2015 16:47:37,847 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.StorageTableResolver - Resolved partitions
for fact fact1: [dt = '2014-06-24-23'] storageTables:[local_fact1]
28 Jul 2015 16:47:37,847 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CubeQueryContext - Not considering fact table
set:[rawfact] as they have non candidate tables and facts missing because of
NO_CANDIDATE_STORAGES
28 Jul 2015 16:47:37,858 [f3b437fb-30fe-4fc1-b265-c5649b80b92b]
[Grizzly-worker(1)] INFO org.apache.lens.server.AuthenticationFilter -
Request from user: null ,
path=/queryapi/queries/53deaaa2-31a6-4f0e-a46f-47f2f00af15e
28 Jul 2015 16:47:37,859 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.LightestFactResolver - Not considering
facts:[fact2] from candidate fact tables as it has more fact weight:200.0
minimum:100.0
28 Jul 2015 16:47:37,859 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CubeQueryContext - Not considering fact
table:fact2 as CandidateTablePruneCause(cause=MORE_WEIGHT, storageCauses=null,
missingPartitions=null, missingUpdatePeriods=null, missingColumns=null,
missingExpressions=null, joinColumns=null, columnsMissingDefaultAggregate=null,
unsupportedTimeDims=null, maxTimeCovered=null, invalidRanges=null)
28 Jul 2015 16:47:37,859 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CubeQueryContext - Available candidate
facts:[[fact1]], picking up [fact1] for querying
28 Jul 2015 16:47:37,859 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO hive.ql.parse.ParseDriver - Parsing expression: (sample_cube.dt =
'2014-06-24-23')
28 Jul 2015 16:47:37,860 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO hive.ql.parse.ParseDriver - Parse Completed
28 Jul 2015 16:47:37,860 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.cube.parse.CubeQueryContext - Picked Fact:[fact1]
dimsToQuery: {}{}
28 Jul 2015 16:47:37,860 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.server.rewrite.RewriteUtil - Final rewritten query for
driver: org.apache.lens.driver.hive.HiveDriver@78c76c3a is: SELECT sum((
sample_cube . measure2 )) FROM local_fact1 sample_cube WHERE (((( sample_cube .
dt ) = '2014-06-24-23' )))
28 Jul 2015 16:47:37,861 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-3]
INFO org.apache.lens.driver.hive.HiveDriver - Estimate: SELECT sum((
sample_cube . measure2 )) FROM local_fact1 sample_cube WHERE (((( sample_cube .
dt ) = '2014-06-24-23' )))
28 Jul 2015 16:47:37,862 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
WARN org.apache.lens.server.rewrite.RewriteUtil - Driver :
org.apache.lens.driver.jdbc.JDBCDriver@3f807a01 Skipped for the query
rewriting due to
org.apache.hadoop.hive.ql.parse.SemanticException: No candidate fact table
available to answer the query, because {"brief":"No candidate storages for any
table","details":{"fact2,fact1,rawfact":[{"cause":"NO_CANDIDATE_STORAGES","storageCauses":{"local":{"cause":"UNSUPPORTED"}}}]}}
at
org.apache.lens.cube.parse.CubeQueryContext.pickCandidateFactToQuery(CubeQueryContext.java:841)
at
org.apache.lens.cube.parse.CubeQueryContext.toHQL(CubeQueryContext.java:852)
at
org.apache.lens.server.rewrite.RewriteUtil$DriverRewriterRunnable.run(RewriteUtil.java:327)
at
org.apache.lens.server.query.QueryExecutionServiceImpl$RewriteEstimateRunnable.run(QueryExecutionServiceImpl.java:1348)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
28 Jul 2015 16:47:37,862 [53deaaa2-31a6-4f0e-a46f-47f2f00af15e] [estimate-4]
ERROR org.apache.lens.server.query.QueryExecutionServiceImpl - Estimate
skipped since rewrite failed for driver
org.apache.lens.driver.jdbc.JDBCDriver@3f807a01 cause: Driver
:org.apache.lens.driver.jdbc.JDBCDriver Cause :No candidate fact table
available to answer the query, because {"brief":"No candidate storages for any
table","details":{"fact2,fact1,rawfact":[{"cause":"NO_CANDIDATE_STORAGES","storageCauses":{"local":{"cause":"UNSUPPORTED"}}}]}}
- Amareshwari Sriramadasu
On July 28, 2015, 11:25 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36884/
> -----------------------------------------------------------
>
> (Updated July 28, 2015, 11:25 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-426
> https://issues.apache.org/jira/browse/LENS-426
>
>
> Repository: lens
>
>
> Description
> -------
>
> Have gone through all the code flow of estimate. Changes include :
>
> - Changing some logs to debug; put isDebugEnabled check for doing costly
> computation on params
> - Have moved to parameterized logging wherever it is not
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryContext.java
> 65c240a
> lens-cube/src/main/java/org/apache/lens/cube/parse/ExpressionResolver.java
> 895cb06
>
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java
> 5fb2441
> lens-cube/src/main/java/org/apache/lens/cube/parse/TimerangeResolver.java
> 820d77d
>
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> 1316257
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java
> 52edffc
>
> Diff: https://reviews.apache.org/r/36884/diff/
>
>
> Testing
> -------
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.069s]
> [INFO] Lens .............................................. SUCCESS [3.084s]
> [INFO] Lens API .......................................... SUCCESS [21.467s]
> [INFO] Lens API for server and extensions ................ SUCCESS [19.744s]
> [INFO] Lens Cube ......................................... SUCCESS [6:02.161s]
> [INFO] Lens DB storage ................................... SUCCESS [18.850s]
> [INFO] Lens Query Library ................................ SUCCESS [15.269s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:45.689s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [35.300s]
> [INFO] Lens Server ....................................... SUCCESS [5:22.277s]
> [INFO] Lens client ....................................... SUCCESS [33.811s]
> [INFO] Lens CLI .......................................... SUCCESS [2:35.257s]
> [INFO] Lens Examples ..................................... SUCCESS [8.561s]
> [INFO] Lens Distribution ................................. SUCCESS [8.048s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:21.468s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.727s]
> [INFO] Lens Regression ................................... SUCCESS [10.513s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 21:06.336s
> [INFO] Finished at: Tue Jul 28 11:17:44 UTC 2015
> [INFO] Final Memory: 175M/1186M
> [INFO]
> ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
