-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36393/
-----------------------------------------------------------
(Updated July 21, 2015, 9:48 a.m.)
Review request for lens.
Changes
-------
Changed implementation of FairPriorityBlockingQueue.
Bugs: LENS-619
https://issues.apache.org/jira/browse/LENS-619
Repository: lens
Description (updated)
-------
Lens will always accept a new query from a user and put it in a scheduling
queue for processing.
Next candidate query picked up from scheduling queue for processing will be
launched only if all query constraints evaluated on candidate query and
launched queries allows the candidate query to be launched, otherwise the
candidate query will be added to waiting queries.
When any launched query is finished, a waiting query selector will select
waiting queries for rescheduling using a list of waiting queries selection
policy. Every waiting query selection policy will return a list of eligible
waiting queries to be rescheduled. Query Selector will calculate intersection
of multiple list of eligible waiting queries and add the result of intersection
to scheduling queue for reprocessing.
At initialization, Query Constraints and Waiting Query Selection Policies will
be configured using configuration values.
New Query Constraints and Waiting Query Selection Policies can be added at
runtime, without rebuilding and deploying lens module. Drivers should be
allowed to add more Query Constraints and Waiting Queries Selection policies.
Waiting Queries will be persisted across server restarts.
Query Constraint 1: Allow a candidate query to be launched, only if, cumulative
query cost of all current queries launched by the user, who submitted candidate
query, is less than a cumulative query cost ceiling for launching a new query.
Waiting Query Selection Policy 1: Select all waiting queries of the user whose
query has finished.
Query Constraint 2: Allow a candidate query to be launched, only if, current
concurrent queries launched on the
selected driver are less than max concurrent queries allowed on the driver.
Waiting Query Selection Policy 2: Select all waiting queries of the driver
whose query has finished.
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/Addable.java
5a556918afcdb8967deee84472cd7863aeafc4fc
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java
f594133e8a75603e5bb0c913d62ea189248cfa05
lens-api/src/main/java/org/apache/lens/api/query/QueryStatus.java
539f0b0b0804257fae6b7ec0af8f2f715175d597
lens-client/src/main/java/org/apache/lens/client/LensStatement.java
f7305fb288167cda088e263a62c77f264d779127
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
7c3e3e08bc7e6949588aa073665fd97596e1a1ae
lens-driver-hive/src/main/resources/hivedriver-default.xml
300ce3fed7814d446b3aeb8688927c2dc8800f81
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java
e3a879929ceb0dad054c71231bc4a70162ecd9ce
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriverConfConstants.java
ebcdaca7c7bab3a15feb10e69ef41238a8bae506
lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
99f6f205dfe6f1c16277ac6624106436df2b2b8f
lens-server-api/pom.xml b0b0fe017a239b24c3b9606b3be77a1722844e0f
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
2b3a4800dc31e1c0787f2da9178456bec871a8a9
lens-server-api/src/main/java/org/apache/lens/server/api/common/ConfigBasedObjectCreationFactory.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java
52207d342ea9ea154ef6ebd1cf4266176039c561
lens-server-api/src/main/java/org/apache/lens/server/api/driver/MinQueryCostSelector.java
c3fe8eb142569c0b254f551c3cbd18502b6eac99
lens-server-api/src/main/java/org/apache/lens/server/api/metrics/MetricsService.java
71128b7d35e91fadf38a9c3926ce46d42369ead4
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
71a37f079f6b43a5cde689ce2c2e8fccb83eb78d
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java
3e7de173a4e3083ca3ac5fa95fd7cd09e7f028bb
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java
6e7ab5dc1c78407a24e389c001ca3100563e3795
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
d72727e7832d02ed2ea90379cd29c7a8e509a8f8
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java
5d7163f112d7c7115ebf8f012ba397bfee63adbc
lens-server-api/src/main/java/org/apache/lens/server/api/query/collect/DriverSpecificWaitingQueriesSelectionPolicy.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/collect/DriverSpecificWaitingQueriesSelectionPolicyFactory.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/collect/EstimatedImmutableQueryCollection.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/collect/ImmutableQueryCollection.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/collect/WaitingQueriesSelectionPolicy.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraintFactory.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/QueryLaunchingConstraint.java
PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/FactPartitionBasedQueryCost.java
7fa5cdb55959a44b52a29884fd629c2479fdfb96
lens-server-api/src/main/java/org/apache/lens/server/api/query/cost/QueryCost.java
be7232134b59f7126edba569358389a9c26a5c74
lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java
f9262994debe0b943123d13c58a22c10b2015e80
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java
dffcfd3567dd015e7c79773c5e8cd6669c2d9d95
lens-server-api/src/test/java/org/apache/lens/server/api/query/collect/DriverSpecificWaitingQueriesSelectionPolicyTest.java
PRE-CREATION
lens-server-api/src/test/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraintTest.java
PRE-CREATION
lens-server-api/src/test/java/org/apache/lens/server/api/query/cost/TestFactPartitionBasedQueryCost.java
0ab0b0ec7c83c1eac8d61c8a30ecec5e9dc2e322
lens-server/pom.xml 6ac30a52cc8c1c58bf8e92bb77df358db6e22a3e
lens-server/src/main/java/org/apache/lens/server/metrics/MetricsServiceImpl.java
39f9f5216acf6e1be7cca6a0851152a78ba66176
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
190435036c5cf8f009d7fbd536ef1743b70c22d9
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
47d27d06272c0c26a93818fdee90cf978a650d10
lens-server/src/main/java/org/apache/lens/server/query/collect/DefaultEstimatedQueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/DefaultQueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/IntersectingWaitingQueriesSelector.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/MutableQueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/QueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/ThreadSafeEstimatedQueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/ThreadSafeQueryCollection.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/UserSpecificWaitingQueriesSelectionPolicy.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/UserSpecificWaitingQueriesSelectionPolicyFactory.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/collect/WaitingQueriesSelector.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/constraint/DefaultQueryLaunchingConstraintsChecker.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/constraint/QueryLaunchingConstraintsChecker.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/constraint/TotalQueryCostCeilingConstraint.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/query/constraint/TotalQueryCostCeilingConstraintFactory.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/util/FairPriorityBlockingQueue.java
PRE-CREATION
lens-server/src/main/java/org/apache/lens/server/util/UtilityMethods.java
715347fab82b5d9d4ad673c45e72900b9a95fc81
lens-server/src/main/resources/lensserver-default.xml
2b540b828a976378bdfbd238e8427a22941fb7e8
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java
7e486aee1e995954bd25beac85a9fafed95ce20e
lens-server/src/test/java/org/apache/lens/server/query/collect/DefaultEstimatedQueryCollectionTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/collect/DefaultQueryCollectionTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/collect/IntersectingWaitingQueriesSelectorTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/collect/QueryCollectUtil.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/collect/ThreadSafetyTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/collect/UserSpecificWaitingQueriesSelectionPolicyTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/constraint/DefaultQueryLaunchingConstraintsCheckerTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/constraint/TotalQueryCostCeilingConstraintFactoryTest.java
PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/constraint/TotalQueryCostCeilingConstraintTest.java
PRE-CREATION
lens-server/src/test/resources/lens-site.xml
3bcbf0e3e44d986a24c00cad84121be72596c64d
lens-server/src/test/resources/log4j.properties
e0f9daa62c7d88f5044803c5a0ce2ffb2ac9315b
src/site/apt/admin/config.apt c5bf70c612a8d9bc383d0031e5cb51338cbd1da8
src/site/apt/admin/hivedriver-config.apt
80a3028789348493a84e753965d32cc30690c503
src/site/apt/admin/jdbcdriver-config.apt
b3d25f6751dd030a4ae818845a4a504ca41d7895
src/site/apt/user/index.apt d48c7347314876989cf8de8f3e330dc972e3ec5d
Diff: https://reviews.apache.org/r/36393/diff/
Testing
-------
Patch rebased with master and all test cases passed:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.312s]
[INFO] Lens .............................................. SUCCESS [4.667s]
[INFO] Lens API .......................................... SUCCESS [31.360s]
[INFO] Lens API for server and extensions ................ SUCCESS [33.701s]
[INFO] Lens Cube ......................................... SUCCESS [5:52.120s]
[INFO] Lens DB storage ................................... SUCCESS [27.322s]
[INFO] Lens Query Library ................................ SUCCESS [22.338s]
[INFO] Lens Hive Driver .................................. SUCCESS [3:44.595s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [52.297s]
[INFO] Lens Server ....................................... SUCCESS [11:28.682s]
[INFO] Lens client ....................................... SUCCESS [46.951s]
[INFO] Lens CLI .......................................... SUCCESS [3:54.445s]
[INFO] Lens Examples ..................................... SUCCESS [13.859s]
[INFO] Lens Distribution ................................. SUCCESS [13.041s]
[INFO] Lens ML Lib ....................................... SUCCESS [2:03.557s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [5.792s]
[INFO] Lens Regression ................................... SUCCESS [15.693s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:34.656s
[INFO] Finished at: Mon Jul 20 13:43:03 IST 2015
[INFO] Final Memory: 123M/336M
[INFO] ------------------------------------------------------------------------
Thanks,
Himanshu Gahlaut
