[jira] [Updated] (LENS-1168) Isolate submissions for each driver
[ https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated LENS-1168: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed myself > Isolate submissions for each driver > --- > > Key: LENS-1168 > URL: https://issues.apache.org/jira/browse/LENS-1168 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 2.6 > > Attachments: LENS-1168.4.patch, LENS-1168.5.patch > > > We need to isolate query submitters for each driver to be different. We have > seen scenarios where hive submission takes time and other interactive queries > on jdbc are queued because querysubmitter was busy. > Also, each driver can have a pool of submitters - which should benefit > interactive queries with immediate submissions, especially the ones submitted > on EXECUTE_WITH_TIMEOUT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LENS-1168) Isolate submissions for each driver
[ https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391341#comment-15391341 ] Hadoop QA commented on LENS-1168: - Applied patch: [LENS-1168.5.patch|https://issues.apache.org/jira/secure/attachment/12819550/LENS-1168.5.patch] and ran command: mvn clean install -fae. Result: Failure. Build Job: https://builds.apache.org/job/PreCommit-Lens-Build/713/ > Isolate submissions for each driver > --- > > Key: LENS-1168 > URL: https://issues.apache.org/jira/browse/LENS-1168 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 2.6 > > Attachments: LENS-1168.4.patch, LENS-1168.5.patch > > > We need to isolate query submitters for each driver to be different. We have > seen scenarios where hive submission takes time and other interactive queries > on jdbc are queued because querysubmitter was busy. > Also, each driver can have a pool of submitters - which should benefit > interactive queries with immediate submissions, especially the ones submitted > on EXECUTE_WITH_TIMEOUT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-128) SchedulerServiceImpl for scheduler.
[ https://issues.apache.org/jira/browse/LENS-128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lavkesh Lahngir updated LENS-128: - Attachment: core-scheduler.patch > SchedulerServiceImpl for scheduler. > --- > > Key: LENS-128 > URL: https://issues.apache.org/jira/browse/LENS-128 > Project: Apache Lens > Issue Type: Sub-task > Components: server >Reporter: Prashant Gupta >Assignee: Lavkesh Lahngir > Fix For: 2.6 > > Attachments: core-scheduler.patch, core-scheduler.patch, > core-scheduler.patch > > > Server APIs will include the support for operations as specified at LENS-124. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (LENS-1227) Support HTTP notification for query STATE changes
[ https://issues.apache.org/jira/browse/LENS-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Puneet Gupta updated LENS-1227: --- Description: 1. HTTP notofication for different stages of query execution -- QUEUED -- RUNNING -- FINISHED 2. Support default info for each state 3. Suppoort extra info for each state 4. Support retries 5. Support enabling notification by defult for a driver type and state (say for hive queries since they take longer and all STATE updates would be nice) 6. Support progress updates in RUNNING state (esp. useful for loing running queries). This will make sense only when progress number is accurate. 7. Support Error count metric for HTTP Notifications was: 1. HTTP notofication for different stages of query execution -- QUEUED -- RUNNING -- FINISHED 2. Support default info for each state 3. Suppoort extra info for each state 4. Support retries 5. Support enabling notification by defult for a driver type and state (say for hive queries since they take longer and all STATE updates would be nice) 6. Support progress updates in RUNNING state (esp. useful for loing running queries). This will make sense only when progress number is accurate. > Support HTTP notification for query STATE changes > - > > Key: LENS-1227 > URL: https://issues.apache.org/jira/browse/LENS-1227 > Project: Apache Lens > Issue Type: New Feature >Reporter: Puneet Gupta >Assignee: Puneet Gupta > > 1. HTTP notofication for different stages of query execution > -- QUEUED > -- RUNNING > -- FINISHED > 2. Support default info for each state > 3. Suppoort extra info for each state > 4. Support retries > 5. Support enabling notification by defult for a driver type and state (say > for hive queries since they take longer and all STATE updates would be nice) > 6. Support progress updates in RUNNING state (esp. useful for loing running > queries). This will make sense only when progress number is accurate. > 7. Support Error count metric for HTTP Notifications -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (LENS-1227) Support HTTP notification for query STATE changes
[ https://issues.apache.org/jira/browse/LENS-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389413#comment-15389413 ] Puneet Gupta edited comment on LENS-1227 at 7/25/16 1:58 AM: - lens.server.query.http.notifictaion.urls = comma separated urls All notifictions will have user, query_id (and query name if any). Apart from this deault info will be shared for each STATE . Extra info can be requested too **lens.query.http.notication.types = (QUEUED,RUNNING,*FINIHSED) **lens.query.http.notication.type.finished.info = (*STATUS,RESULT,TOTALRUNTIME,TOTALWAIT,etc) **lens.query.http.notication.type.queued.info = (*QUEUENUMBER,PRIORITY,etc) **lens.query.http.notication.type.running.info = (TOTALWAIT,etc) **lens.query.http.notifictaion.retries = 2 (Default is 0) **= not mandatory *= Default option was (Author: puneet884): lens.server.query.http.notifictaion.urls = comma separated urls All notifictions will have user, query_id (and query name if any). Apart from this deault info will be shared for each STATE . Extra info can be requested too **lens.server.query.http.notication.types = (QUEUED,RUNNING,*FINIHSED) **lens.server.query.http.notication.type.finished.info = (*STATUS,RESULT,TOTALRUNTIME,TOTALWAIT,etc) **lens.server.query.http.notication.type.queued.info = (*QUEUENUMBER,PRIORITY,etc) **lens.server.query.http.notication.type.running.info = (TOTALWAIT,etc) **lens.server.query.http.notifictaion.retries = 2 (Default is 0) **= not mandatory *= Default option > Support HTTP notification for query STATE changes > - > > Key: LENS-1227 > URL: https://issues.apache.org/jira/browse/LENS-1227 > Project: Apache Lens > Issue Type: New Feature >Reporter: Puneet Gupta >Assignee: Puneet Gupta > > 1. HTTP notofication for different stages of query execution > -- QUEUED > -- RUNNING > -- FINISHED > 2. Support default info for each state > 3. Suppoort extra info for each state > 4. Support retries > 5. Support enabling notification by defult for a driver type and state (say > for hive queries since they take longer and all STATE updates would be nice) > 6. Support progress updates in RUNNING state (esp. useful for loing running > queries). This will make sense only when progress number is accurate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (LENS-1227) Support HTTP notification for query STATE changes
[ https://issues.apache.org/jira/browse/LENS-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389413#comment-15389413 ] Puneet Gupta edited comment on LENS-1227 at 7/25/16 1:58 AM: - lens.query.http.notifictaion.urls = comma separated urls All notifictions will have user, query_id (and query name if any). Apart from this deault info will be shared for each STATE . Extra info can be requested too **lens.query.http.notication.types = (QUEUED,RUNNING,*FINIHSED) **lens.query.http.notication.type.finished.info = (*STATUS,RESULT,TOTALRUNTIME,TOTALWAIT,etc) **lens.query.http.notication.type.queued.info = (*QUEUENUMBER,PRIORITY,etc) **lens.query.http.notication.type.running.info = (TOTALWAIT,etc) **lens.query.http.notifictaion.retries = 2 (Default is 0) **= not mandatory *= Default option was (Author: puneet884): lens.server.query.http.notifictaion.urls = comma separated urls All notifictions will have user, query_id (and query name if any). Apart from this deault info will be shared for each STATE . Extra info can be requested too **lens.query.http.notication.types = (QUEUED,RUNNING,*FINIHSED) **lens.query.http.notication.type.finished.info = (*STATUS,RESULT,TOTALRUNTIME,TOTALWAIT,etc) **lens.query.http.notication.type.queued.info = (*QUEUENUMBER,PRIORITY,etc) **lens.query.http.notication.type.running.info = (TOTALWAIT,etc) **lens.query.http.notifictaion.retries = 2 (Default is 0) **= not mandatory *= Default option > Support HTTP notification for query STATE changes > - > > Key: LENS-1227 > URL: https://issues.apache.org/jira/browse/LENS-1227 > Project: Apache Lens > Issue Type: New Feature >Reporter: Puneet Gupta >Assignee: Puneet Gupta > > 1. HTTP notofication for different stages of query execution > -- QUEUED > -- RUNNING > -- FINISHED > 2. Support default info for each state > 3. Suppoort extra info for each state > 4. Support retries > 5. Support enabling notification by defult for a driver type and state (say > for hive queries since they take longer and all STATE updates would be nice) > 6. Support progress updates in RUNNING state (esp. useful for loing running > queries). This will make sense only when progress number is accurate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 49371: LENS-1168 : Add query launcher pool
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49371/#review143343 --- Ship it! Ship It! - Puneet Gupta On July 22, 2016, 4:05 a.m., Amareshwari Sriramadasu wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49371/ > --- > > (Updated July 22, 2016, 4:05 a.m.) > > > Review request for lens. > > > Bugs: LENS-1168 > https://issues.apache.org/jira/browse/LENS-1168 > > > Repository: lens > > > Description > --- > > Changes include : > - Add query launcher pool which would launch queries after applying > constraint checks. > - Add max concurrent launching queries constraint - to limit number parallel > launches at any time. > - Update StatusPoller to skip launching queries > - Update cancelQuery to interrupt launcher. > - Added test to verify parallel launches. > > > Diffs > - > > > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java > bd9b1ab > > lens-server-api/src/main/java/org/apache/lens/server/api/metrics/MetricsService.java > 7fd2d81 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java > 8ba0689 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java > d10ad09 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java > b2319a9 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraintFactory.java > 442cd99 > > lens-server-api/src/test/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraintTest.java > 38b74ae > > lens-server/src/main/java/org/apache/lens/server/metrics/MetricsServiceImpl.java > 1e8d540 > > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java > be0e2c8 > > lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java > e77c7fa > lens-server/src/main/resources/lensserver-default.xml 3ce8fc1 > lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java > d5e975a > > lens-server/src/test/java/org/apache/lens/server/common/FailingQueryDriver.java > 0b38517 > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryConstraints.java > 0df436a > > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java > 6af4225 > > lens-server/src/test/java/org/apache/lens/server/query/constraint/ThreadSafeEstimatedQueryCollectionTest.java > c8ebd0c > lens-server/src/test/resources/logback.xml a345054 > src/site/apt/admin/config.apt 852955b > > Diff: https://reviews.apache.org/r/49371/diff/ > > > Testing > --- > > > Thanks, > > Amareshwari Sriramadasu > >
[jira] [Commented] (LENS-1168) Isolate submissions for each driver
[ https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391215#comment-15391215 ] Hadoop QA commented on LENS-1168: - Applied patch: [LENS-1168.5.patch|https://issues.apache.org/jira/secure/attachment/12819550/LENS-1168.5.patch] and ran command: mvn clean install -fae. Result: Failure. Build Job: https://builds.apache.org/job/PreCommit-Lens-Build/712/ > Isolate submissions for each driver > --- > > Key: LENS-1168 > URL: https://issues.apache.org/jira/browse/LENS-1168 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 2.6 > > Attachments: LENS-1168.4.patch, LENS-1168.5.patch > > > We need to isolate query submitters for each driver to be different. We have > seen scenarios where hive submission takes time and other interactive queries > on jdbc are queued because querysubmitter was busy. > Also, each driver can have a pool of submitters - which should benefit > interactive queries with immediate submissions, especially the ones submitted > on EXECUTE_WITH_TIMEOUT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LENS-1168) Isolate submissions for each driver
[ https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391212#comment-15391212 ] Hadoop QA commented on LENS-1168: - Applied patch: [LENS-1168.5.patch|https://issues.apache.org/jira/secure/attachment/12819550/LENS-1168.5.patch] and ran command: mvn clean install -fae. Result: Failure. Build Job: https://builds.apache.org/job/PreCommit-Lens-Build/711/ > Isolate submissions for each driver > --- > > Key: LENS-1168 > URL: https://issues.apache.org/jira/browse/LENS-1168 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 2.6 > > Attachments: LENS-1168.4.patch, LENS-1168.5.patch > > > We need to isolate query submitters for each driver to be different. We have > seen scenarios where hive submission takes time and other interactive queries > on jdbc are queued because querysubmitter was busy. > Also, each driver can have a pool of submitters - which should benefit > interactive queries with immediate submissions, especially the ones submitted > on EXECUTE_WITH_TIMEOUT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
