[jira] [Closed] (AURORA-1817) Implement ReservationStore
[ https://issues.apache.org/jira/browse/AURORA-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko closed AURORA-1817. -- Resolution: Invalid > Implement ReservationStore > -- > > Key: AURORA-1817 > URL: https://issues.apache.org/jira/browse/AURORA-1817 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > > A new store that will keep state of tasks that need reserved resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (AURORA-1818) Allow jobs to request dynamically reserved resources
[ https://issues.apache.org/jira/browse/AURORA-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko closed AURORA-1818. -- Resolution: Invalid > Allow jobs to request dynamically reserved resources > > > Key: AURORA-1818 > URL: https://issues.apache.org/jira/browse/AURORA-1818 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > > Wire up configuration changes that allow a task to be dynamically reserved > via the API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (AURORA-1822) Change scheduling logic to have DR preempt jobs if timer waiting for reservation to come back has expired
[ https://issues.apache.org/jira/browse/AURORA-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko closed AURORA-1822. -- Resolution: Invalid > Change scheduling logic to have DR preempt jobs if timer waiting for > reservation to come back has expired > - > > Key: AURORA-1822 > URL: https://issues.apache.org/jira/browse/AURORA-1822 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > > AURORA-1819 will not preempt existing tasks if no offer is available. We need > ability to have jobs needing dynamic reservations ability to preempt. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AURORA-1735) Enabling jobs to dynamically reserve cluster resources through Aurora
[ https://issues.apache.org/jira/browse/AURORA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075497#comment-16075497 ] Dmitriy Shirchenko commented on AURORA-1735: Code is here (with a couple of orthogonal patches): reviews.apache.org/r/57487/ > Enabling jobs to dynamically reserve cluster resources through Aurora > - > > Key: AURORA-1735 > URL: https://issues.apache.org/jira/browse/AURORA-1735 > Project: Aurora > Issue Type: Epic > Components: Scheduler >Reporter: Mehrdad Nurolahzade >Assignee: Dmitriy Shirchenko > > As a precursor to support persistent services through Apache Aurora we have > to enable dynamic reservation of resources through Aurora. A stateful service > needs the ability to create a persistent disk volume, and the ability to > reserve resources. A persistent volume is necessary to actually store the > state, and reservation of resources is necessary in order to reliably > re-offer the service: the state itself, along with the resources necessary to > retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take > advantage of these new features that are now supported by Mesos. > RFC document: > [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] > Design Doc: > [https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AURORA-1735) Enabling jobs to dynamically reserve cluster resources through Aurora
[ https://issues.apache.org/jira/browse/AURORA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075453#comment-16075453 ] Dmitriy Shirchenko commented on AURORA-1735: [~rdelvalle] This was successfully deployed internally. I can update the RB with the code if you are interested. > Enabling jobs to dynamically reserve cluster resources through Aurora > - > > Key: AURORA-1735 > URL: https://issues.apache.org/jira/browse/AURORA-1735 > Project: Aurora > Issue Type: Epic > Components: Scheduler >Reporter: Mehrdad Nurolahzade >Assignee: Dmitriy Shirchenko > > As a precursor to support persistent services through Apache Aurora we have > to enable dynamic reservation of resources through Aurora. A stateful service > needs the ability to create a persistent disk volume, and the ability to > reserve resources. A persistent volume is necessary to actually store the > state, and reservation of resources is necessary in order to reliably > re-offer the service: the state itself, along with the resources necessary to > retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take > advantage of these new features that are now supported by Mesos. > RFC document: > [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] > Design Doc: > [https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AURORA-1881) Capture stderr from output for shell health check failures.
Dmitriy Shirchenko created AURORA-1881: -- Summary: Capture stderr from output for shell health check failures. Key: AURORA-1881 URL: https://issues.apache.org/jira/browse/AURORA-1881 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko existing `check_call` doesn't return errors output by the process [1], so we need to plumb that through back to the user via `check_output` [2] [1] https://github.com/google/python-subprocess32/blob/master/subprocess32.py#L602 [2] https://github.com/google/python-subprocess32/blob/master/subprocess32.py#L638 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AURORA-1819) Scheduling logic changes to support dynamic reservations
[ https://issues.apache.org/jira/browse/AURORA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko reassigned AURORA-1819: -- Assignee: Dmitriy Shirchenko > Scheduling logic changes to support dynamic reservations > > > Key: AURORA-1819 > URL: https://issues.apache.org/jira/browse/AURORA-1819 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > > TaskAssigner and OfferManager will need to be modified to support dynamic > reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1816) Implement Offer Reconciler
[ https://issues.apache.org/jira/browse/AURORA-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802865#comment-15802865 ] Dmitriy Shirchenko commented on AURORA-1816: Draft here which will be released along with the rest of the subtasks of AURORA-1785 https://github.com/apache/aurora/compare/master...shirchen:AURORA-1816 > Implement Offer Reconciler > --- > > Key: AURORA-1816 > URL: https://issues.apache.org/jira/browse/AURORA-1816 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > > A new service that will unreserve resources which have been previously > reserved but are no longer needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1822) Change scheduling logic to have DR preempt jobs if timer waiting for reservation to come back has expired
Dmitriy Shirchenko created AURORA-1822: -- Summary: Change scheduling logic to have DR preempt jobs if timer waiting for reservation to come back has expired Key: AURORA-1822 URL: https://issues.apache.org/jira/browse/AURORA-1822 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko AURORA-1819 will not preempt existing tasks if no offer is available. We need ability to have jobs needing dynamic reservations ability to preempt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1818) Allow jobs to request dynamically reserved resources
Dmitriy Shirchenko created AURORA-1818: -- Summary: Allow jobs to request dynamically reserved resources Key: AURORA-1818 URL: https://issues.apache.org/jira/browse/AURORA-1818 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko Wire up configuration changes that allow a task to be dynamically reserved via the API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1817) Implement ReservationStore
Dmitriy Shirchenko created AURORA-1817: -- Summary: Implement ReservationStore Key: AURORA-1817 URL: https://issues.apache.org/jira/browse/AURORA-1817 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko A new store that will keep state of tasks that need reserved resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1735) Enabling jobs to dynamically reserve cluster resources through Aurora
[ https://issues.apache.org/jira/browse/AURORA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1735: --- Description: As a precursor to support persistent services through Apache Aurora we have to enable dynamic reservation of resources through Aurora. A stateful service needs the ability to create a persistent disk volume, and the ability to reserve resources. A persistent volume is necessary to actually store the state, and reservation of resources is necessary in order to reliably re-offer the service: the state itself, along with the resources necessary to retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take advantage of these new features that are now supported by Mesos. RFC document: [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] Design Doc: [https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE/edit#heading=h.klg3urfbnq3v] was: As a precursor to support persistent services through Apache Aurora we have to enable dynamic reservation of resources through Aurora. A stateful service needs the ability to create a persistent disk volume, and the ability to reserve resources. A persistent volume is necessary to actually store the state, and reservation of resources is necessary in order to reliably re-offer the service: the state itself, along with the resources necessary to retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take advantage of these new features that are now supported by Mesos. RFC document: [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] Design Doc: [https://docs.google.com/document/d/19gV8Po6DIHO14tOC7Qouk8RnboY8UCfRTninwn_5-7c/edit] > Enabling jobs to dynamically reserve cluster resources through Aurora > - > > Key: AURORA-1735 > URL: https://issues.apache.org/jira/browse/AURORA-1735 > Project: Aurora > Issue Type: Epic > Components: Scheduler >Reporter: Mehrdad Nurolahzade >Assignee: Dmitriy Shirchenko > > As a precursor to support persistent services through Apache Aurora we have > to enable dynamic reservation of resources through Aurora. A stateful service > needs the ability to create a persistent disk volume, and the ability to > reserve resources. A persistent volume is necessary to actually store the > state, and reservation of resources is necessary in order to reliably > re-offer the service: the state itself, along with the resources necessary to > retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take > advantage of these new features that are now supported by Mesos. > RFC document: > [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] > Design Doc: > [https://docs.google.com/document/d/1L2EKEcKKBPmuxRviSUebyuqiNwaO-2hsITBjt3SgWvE/edit#heading=h.klg3urfbnq3v] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AURORA-1735) Enabling jobs to dynamically reserve cluster resources through Aurora
[ https://issues.apache.org/jira/browse/AURORA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko reassigned AURORA-1735: -- Assignee: Dmitriy Shirchenko > Enabling jobs to dynamically reserve cluster resources through Aurora > - > > Key: AURORA-1735 > URL: https://issues.apache.org/jira/browse/AURORA-1735 > Project: Aurora > Issue Type: Epic > Components: Scheduler >Reporter: Mehrdad Nurolahzade >Assignee: Dmitriy Shirchenko > > As a precursor to support persistent services through Apache Aurora we have > to enable dynamic reservation of resources through Aurora. A stateful service > needs the ability to create a persistent disk volume, and the ability to > reserve resources. A persistent volume is necessary to actually store the > state, and reservation of resources is necessary in order to reliably > re-offer the service: the state itself, along with the resources necessary to > retrieve the state (e.g. cpu and memory). Aurora needs to be extended to take > advantage of these new features that are now supported by Mesos. > RFC document: > [https://docs.google.com/document/d/15n29HSQPXuFrnxZAgfVINTRP1Iv47_jfcstJNuMwr5A] > Design Doc: > [https://docs.google.com/document/d/19gV8Po6DIHO14tOC7Qouk8RnboY8UCfRTninwn_5-7c/edit] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1769) Enabling webhook is synchronous and could cause longer leader reelection cycle
[ https://issues.apache.org/jira/browse/AURORA-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1769: --- Assignee: (was: Dmitriy Shirchenko) > Enabling webhook is synchronous and could cause longer leader reelection cycle > -- > > Key: AURORA-1769 > URL: https://issues.apache.org/jira/browse/AURORA-1769 > Project: Aurora > Issue Type: Bug >Reporter: Dmitriy Shirchenko > > We had an issue where on scheduler leader reelection EventBus was full of > TaskStateChange events and caused scheduler to not be able to post > DriverRegistered() message which caused Aurora scheduler to not register > within 1 minute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1773) Asynchronously call webhook endpoint not to block EventBus
[ https://issues.apache.org/jira/browse/AURORA-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1773: --- Assignee: (was: Dmitriy Shirchenko) > Asynchronously call webhook endpoint not to block EventBus > -- > > Key: AURORA-1773 > URL: https://issues.apache.org/jira/browse/AURORA-1773 > Project: Aurora > Issue Type: Sub-task >Reporter: Dmitriy Shirchenko > Labels: uber > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (AURORA-1776) Switch `launchTask` Mesos Driver call in favor of `acceptOffers` to support future dynamic reservation operations
[ https://issues.apache.org/jira/browse/AURORA-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko closed AURORA-1776. -- > Switch `launchTask` Mesos Driver call in favor of `acceptOffers` to support > future dynamic reservation operations > - > > Key: AURORA-1776 > URL: https://issues.apache.org/jira/browse/AURORA-1776 > Project: Aurora > Issue Type: Sub-task > Components: Scheduler >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > Labels: uber > Fix For: 0.16.0 > > > Dynamic reservations cannot be launched via `launchTask` Mesos driver call so > we need to switch to using `acceptOffers`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1783) Probable http connection leak
[ https://issues.apache.org/jira/browse/AURORA-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1783: --- Fix Version/s: 0.17.0 > Probable http connection leak > - > > Key: AURORA-1783 > URL: https://issues.apache.org/jira/browse/AURORA-1783 > Project: Aurora > Issue Type: Bug >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > Fix For: 0.17.0 > > > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.lang.Thread.run(Thread.java:745) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > com.google.common.eventbus.Subscriber$1.run(Subscriber.java:80) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:154) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:95) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > java.lang.reflect.Method.invoke(Method.java:498) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.aurora.scheduler.events.Webhook.taskChangedState(Webhook.java:77) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: at > org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286) > Sep 26 19:34:12 compute59-sjc1 aurora-scheduler[25557]: E0926 19:34:12.608 > [AsyncProcessor-4, Webhook:79] Error sending a Webhook event > org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for > connection from pool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1776) Switch `launchTask` Mesos Driver call in favor of `acceptOffers` to support future dynamic reservation operations
Dmitriy Shirchenko created AURORA-1776: -- Summary: Switch `launchTask` Mesos Driver call in favor of `acceptOffers` to support future dynamic reservation operations Key: AURORA-1776 URL: https://issues.apache.org/jira/browse/AURORA-1776 Project: Aurora Issue Type: Sub-task Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko Dynamic reservations cannot be launched via `launchTask` Mesos driver call so we need to switch to using `acceptOffers`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1772) Do not resend TaskStateChange events every time a scheduler starts
[ https://issues.apache.org/jira/browse/AURORA-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498106#comment-15498106 ] Dmitriy Shirchenko commented on AURORA-1772: https://reviews.apache.org/r/51980/ > Do not resend TaskStateChange events every time a scheduler starts > -- > > Key: AURORA-1772 > URL: https://issues.apache.org/jira/browse/AURORA-1772 > Project: Aurora > Issue Type: Sub-task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > Labels: uber > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1773) Asynchronously call webhook endpoint not to block EventBus
Dmitriy Shirchenko created AURORA-1773: -- Summary: Asynchronously call webhook endpoint not to block EventBus Key: AURORA-1773 URL: https://issues.apache.org/jira/browse/AURORA-1773 Project: Aurora Issue Type: Sub-task Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1772) Do not resend TaskStateChange events every time a scheduler starts
Dmitriy Shirchenko created AURORA-1772: -- Summary: Do not resend TaskStateChange events every time a scheduler starts Key: AURORA-1772 URL: https://issues.apache.org/jira/browse/AURORA-1772 Project: Aurora Issue Type: Sub-task Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1769) Enabling webhook is synchronous and could cause longer leader reelection cycle
Dmitriy Shirchenko created AURORA-1769: -- Summary: Enabling webhook is synchronous and could cause longer leader reelection cycle Key: AURORA-1769 URL: https://issues.apache.org/jira/browse/AURORA-1769 Project: Aurora Issue Type: Bug Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko We had an issue where on scheduler leader reelection EventBus was full of TaskStateChange events and caused scheduler to not be able to post DriverRegistered() message which caused Aurora scheduler to not register within 1 minute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1733) Introduce a TaskHistory struct to TaskQuery to support limiting number of taskEvents per instance
Dmitriy Shirchenko created AURORA-1733:
--
Summary: Introduce a TaskHistory struct to TaskQuery to support
limiting number of taskEvents per instance
Key: AURORA-1733
URL: https://issues.apache.org/jira/browse/AURORA-1733
Project: Aurora
Issue Type: Story
Reporter: Dmitriy Shirchenko
Assignee: Maxim Khutornenko
Our use case is to pull back latest task state history but we only want to pull
back a specific number of events per instance (usually 1). We have not been
able to find a way to do that.
Right now we use TaskQuery with getTasksWithoutConfigs with a TaskQuery that
pulls back everything in a healthy state (includes ASSIGNED, RUNNING) to
display to our users on our internal UI. This has become too big of a blob to
process.
Proposal is to introduce and implement the following:
{code}
struct TaskHistory {
0: set statuses
1: i32 limit
}
{code}
and add it to TaskQuery
{code}
struct TaskQuery {
...
15: TaskHistory taskHistory
}
{code}
This is an alternative iteration on proposal in AURORA-1722 and are related
tasks.
Comments?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1721) Support user initiated rollback
[ https://issues.apache.org/jira/browse/AURORA-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345644#comment-15345644 ] Dmitriy Shirchenko commented on AURORA-1721: Can someone please reassign this to Igor? He is interested in working on this task from our side! Thanks! > Support user initiated rollback > > > Key: AURORA-1721 > URL: https://issues.apache.org/jira/browse/AURORA-1721 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: Igor Morozov >Assignee: Dmitriy Shirchenko > Labels: Uber > Fix For: 0.16.0 > > > The proposal to support user initiated rollback: > 1. Create new thrift API: > /**Rollback job update. */ > Response rollbackJobUpdate( > /** The update to rollback. */ > 1: JobUpdateKey key, > /** A user-specified message to include with the induced job update > state change. */ > 3: string message) > 2. Implement new API in a scheduler so the implementation would just undo > the latest JobUpdate effectively trying to apply initialState to the job. If > that is for some reason is impossible them rollback with fail with > appropriate error message. > 3. Support new aurora client command 'rollback' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1721) Support user initiated rollback
[ https://issues.apache.org/jira/browse/AURORA-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1721: --- Labels: Uber (was: ) > Support user initiated rollback > > > Key: AURORA-1721 > URL: https://issues.apache.org/jira/browse/AURORA-1721 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: Igor Morozov >Assignee: Dmitriy Shirchenko > Labels: Uber > Fix For: 0.16.0 > > > The proposal to support user initiated rollback: > 1. Create new thrift API: > /**Rollback job update. */ > Response rollbackJobUpdate( > /** The update to rollback. */ > 1: JobUpdateKey key, > /** A user-specified message to include with the induced job update > state change. */ > 3: string message) > 2. Implement new API in a scheduler so the implementation would just undo > the latest JobUpdate effectively trying to apply initialState to the job. If > that is for some reason is impossible them rollback with fail with > appropriate error message. > 3. Support new aurora client command 'rollback' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AURORA-1721) Support user initiated rollback
[ https://issues.apache.org/jira/browse/AURORA-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko reassigned AURORA-1721: -- Assignee: Dmitriy Shirchenko > Support user initiated rollback > > > Key: AURORA-1721 > URL: https://issues.apache.org/jira/browse/AURORA-1721 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: Igor Morozov >Assignee: Dmitriy Shirchenko > Fix For: 0.16.0 > > > The proposal to support user initiated rollback: > 1. Create new thrift API: > /**Rollback job update. */ > Response rollbackJobUpdate( > /** The update to rollback. */ > 1: JobUpdateKey key, > /** A user-specified message to include with the induced job update > state change. */ > 3: string message) > 2. Implement new API in a scheduler so the implementation would just undo > the latest JobUpdate effectively trying to apply initialState to the job. If > that is for some reason is impossible them rollback with fail with > appropriate error message. > 3. Support new aurora client command 'rollback' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1709) After rb/46835 job key's role is required to exist as a user
Dmitriy Shirchenko created AURORA-1709: -- Summary: After rb/46835 job key's role is required to exist as a user Key: AURORA-1709 URL: https://issues.apache.org/jira/browse/AURORA-1709 Project: Aurora Issue Type: Bug Reporter: Dmitriy Shirchenko Assignee: Joshua Cohen sandbox.py had the change eg ``` return FileSystemImageSandbox(self.SANDBOX_NAME, self._get_sandbox_user(assigned_task)) ``` which is a problem for us since ``` pwent = pwd.getpwnam(self._user) grent = grp.getgrgid(pwent.pw_gid) ``` throw an exception if the user does not exist. This is a change in behavior of how Aurora launched containers before the diff. Before, job key's role could be mangled liberally and arbitrarily w/out this restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1683) Implement an optional webhook parameter to scheduler to POST TaskStateChange events
[ https://issues.apache.org/jira/browse/AURORA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1683: --- Fix Version/s: 0.14.0 > Implement an optional webhook parameter to scheduler to POST TaskStateChange > events > --- > > Key: AURORA-1683 > URL: https://issues.apache.org/jira/browse/AURORA-1683 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > Fix For: 0.14.0 > > > Per discussion on mailing list, I propose adding a webhook to publish > TaskStateChange events to an HTTP REST point. This will be an optional flag > passed into scheduler with an optional configuration file that will contain > optional headers that need to be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1683) Implement an optional webhook parameter to scheduler to POST TaskStateChange events
[ https://issues.apache.org/jira/browse/AURORA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1683: --- Description: Per discussion on mailing list, I propose adding a webhook to publish TaskStateChange events to an HTTP REST point. This will be an optional flag passed into scheduler with an optional configuration file that will contain optional headers that need to be set. (was: Per discussion on mailing list, I propose adding a webhook to publish ) > Implement an optional webhook parameter to scheduler to POST TaskStateChange > events > --- > > Key: AURORA-1683 > URL: https://issues.apache.org/jira/browse/AURORA-1683 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > > Per discussion on mailing list, I propose adding a webhook to publish > TaskStateChange events to an HTTP REST point. This will be an optional flag > passed into scheduler with an optional configuration file that will contain > optional headers that need to be set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1683) Implement an optional webhook parameter to scheduler to POST TaskStateChange events
[ https://issues.apache.org/jira/browse/AURORA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1683: --- Description: Per discussion on mailing list, I propose adding a webhook to publish > Implement an optional webhook parameter to scheduler to POST TaskStateChange > events > --- > > Key: AURORA-1683 > URL: https://issues.apache.org/jira/browse/AURORA-1683 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > > Per discussion on mailing list, I propose adding a webhook to publish -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AURORA-1683) Implement an optional webhook parameter to scheduler to POST TaskStateChange events
[ https://issues.apache.org/jira/browse/AURORA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko reassigned AURORA-1683: -- Assignee: Dmitriy Shirchenko > Implement an optional webhook parameter to scheduler to POST TaskStateChange > events > --- > > Key: AURORA-1683 > URL: https://issues.apache.org/jira/browse/AURORA-1683 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1683) Implement an optional webhook parameter to scheduler to POST TaskStateChange events
[ https://issues.apache.org/jira/browse/AURORA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1683: --- Summary: Implement an optional webhook parameter to scheduler to POST TaskStateChange events (was: Pointers on how to build a subscriber that could send TaskStateChange events to a Kafka topic) > Implement an optional webhook parameter to scheduler to POST TaskStateChange > events > --- > > Key: AURORA-1683 > URL: https://issues.apache.org/jira/browse/AURORA-1683 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1683) Pointers on how to build a subscriber that could send TaskStateChange events to a Kafka topic
Dmitriy Shirchenko created AURORA-1683: -- Summary: Pointers on how to build a subscriber that could send TaskStateChange events to a Kafka topic Key: AURORA-1683 URL: https://issues.apache.org/jira/browse/AURORA-1683 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (AURORA-1666) Allow health check to not run as job's role `role`
[
https://issues.apache.org/jira/browse/AURORA-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Shirchenko reopened AURORA-1666:
Previous RB was done incorrectly, as a boolean was not passed in.
```
thermos_executor.pex: error: --setuid-health-checks option does not take a value
```
> Allow health check to not run as job's role `role`
> --
>
> Key: AURORA-1666
> URL: https://issues.apache.org/jira/browse/AURORA-1666
> Project: Aurora
> Issue Type: Story
>Reporter: Dmitriy Shirchenko
>Assignee: Dmitriy Shirchenko
>
> We have an issue with new permission demotion as our containers run as the
> same user and we don't care what user they run as.
> I propose a flag to disable this demotion and allow health check to keep
> running as root if we want this. Maybe this can be done at thermos executor
> level.
> {code}
> E0415 20:32:51.799864 6 aurora_executor.py:118] Traceback (most recent call
> last):
> File "apache/aurora/executor/aurora_executor.py", line 116, in _run
> self._start_status_manager(driver, assigned_task)
> File "apache/aurora/executor/aurora_executor.py", line 161, in
> _start_status_manager
> status_checker = status_provider.from_assigned_task(assigned_task,
> self._sandbox)
> File "apache/aurora/executor/common/health_checker.py", line 248, in
> from_assigned_task
> pw_entry = pwd.getpwnam(assigned_task.task.job.role)
> KeyError: 'getpwnam(): name not found: meta-umonitor-worker'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1666) Allow health check to not run as job's role `role`
[
https://issues.apache.org/jira/browse/AURORA-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246974#comment-15246974
]
Dmitriy Shirchenko edited comment on AURORA-1666 at 4/19/16 2:03 AM:
-
Previous RB was done incorrectly, as a boolean was not passed in.
{code}
thermos_executor.pex: error: --setuid-health-checks option does not take a value
{code}
was (Author: shirchen):
Previous RB was done incorrectly, as a boolean was not passed in.
```
thermos_executor.pex: error: --setuid-health-checks option does not take a value
```
> Allow health check to not run as job's role `role`
> --
>
> Key: AURORA-1666
> URL: https://issues.apache.org/jira/browse/AURORA-1666
> Project: Aurora
> Issue Type: Story
>Reporter: Dmitriy Shirchenko
>Assignee: Dmitriy Shirchenko
>
> We have an issue with new permission demotion as our containers run as the
> same user and we don't care what user they run as.
> I propose a flag to disable this demotion and allow health check to keep
> running as root if we want this. Maybe this can be done at thermos executor
> level.
> {code}
> E0415 20:32:51.799864 6 aurora_executor.py:118] Traceback (most recent call
> last):
> File "apache/aurora/executor/aurora_executor.py", line 116, in _run
> self._start_status_manager(driver, assigned_task)
> File "apache/aurora/executor/aurora_executor.py", line 161, in
> _start_status_manager
> status_checker = status_provider.from_assigned_task(assigned_task,
> self._sandbox)
> File "apache/aurora/executor/common/health_checker.py", line 248, in
> from_assigned_task
> pw_entry = pwd.getpwnam(assigned_task.task.job.role)
> KeyError: 'getpwnam(): name not found: meta-umonitor-worker'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1666) Allow health check to not run as job's role `role`
[
https://issues.apache.org/jira/browse/AURORA-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243845#comment-15243845
]
Dmitriy Shirchenko commented on AURORA-1666:
https://reviews.apache.org/r/46290/
> Allow health check to not run as job's role `role`
> --
>
> Key: AURORA-1666
> URL: https://issues.apache.org/jira/browse/AURORA-1666
> Project: Aurora
> Issue Type: Story
>Reporter: Dmitriy Shirchenko
>Assignee: Dmitriy Shirchenko
>
> We have an issue with new permission demotion as our containers run as the
> same user and we don't care what user they run as.
> I propose a flag to disable this demotion and allow health check to keep
> running as root if we want this. Maybe this can be done at thermos executor
> level.
> {code}
> E0415 20:32:51.799864 6 aurora_executor.py:118] Traceback (most recent call
> last):
> File "apache/aurora/executor/aurora_executor.py", line 116, in _run
> self._start_status_manager(driver, assigned_task)
> File "apache/aurora/executor/aurora_executor.py", line 161, in
> _start_status_manager
> status_checker = status_provider.from_assigned_task(assigned_task,
> self._sandbox)
> File "apache/aurora/executor/common/health_checker.py", line 248, in
> from_assigned_task
> pw_entry = pwd.getpwnam(assigned_task.task.job.role)
> KeyError: 'getpwnam(): name not found: meta-umonitor-worker'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1666) Allow health check to not run as job's role `role`
[
https://issues.apache.org/jira/browse/AURORA-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243736#comment-15243736
]
Dmitriy Shirchenko commented on AURORA-1666:
Sounds [~zmanji] is onboard with suggestion of adding a thermos_executor flag.
> Allow health check to not run as job's role `role`
> --
>
> Key: AURORA-1666
> URL: https://issues.apache.org/jira/browse/AURORA-1666
> Project: Aurora
> Issue Type: Story
>Reporter: Dmitriy Shirchenko
>Assignee: Dmitriy Shirchenko
>
> We have an issue with new permission demotion as our containers run as the
> same user and we don't care what user they run as.
> I propose a flag to disable this demotion and allow health check to keep
> running as root if we want this. Maybe this can be done at thermos executor
> level.
> {code}
> E0415 20:32:51.799864 6 aurora_executor.py:118] Traceback (most recent call
> last):
> File "apache/aurora/executor/aurora_executor.py", line 116, in _run
> self._start_status_manager(driver, assigned_task)
> File "apache/aurora/executor/aurora_executor.py", line 161, in
> _start_status_manager
> status_checker = status_provider.from_assigned_task(assigned_task,
> self._sandbox)
> File "apache/aurora/executor/common/health_checker.py", line 248, in
> from_assigned_task
> pw_entry = pwd.getpwnam(assigned_task.task.job.role)
> KeyError: 'getpwnam(): name not found: meta-umonitor-worker'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AURORA-1666) Allow health check to not run as job's role `role`
Dmitriy Shirchenko created AURORA-1666:
--
Summary: Allow health check to not run as job's role `role`
Key: AURORA-1666
URL: https://issues.apache.org/jira/browse/AURORA-1666
Project: Aurora
Issue Type: Story
Reporter: Dmitriy Shirchenko
We have an issue with new permission demotion as our containers run as the same
user and we don't care what user they run as.
I propose a flag to disable this demotion and allow health check to keep
running as root if we want this. Maybe this can be done at thermos executor
level.
{code}
E0415 20:32:51.799864 6 aurora_executor.py:118] Traceback (most recent call
last):
File "apache/aurora/executor/aurora_executor.py", line 116, in _run
self._start_status_manager(driver, assigned_task)
File "apache/aurora/executor/aurora_executor.py", line 161, in
_start_status_manager
status_checker = status_provider.from_assigned_task(assigned_task,
self._sandbox)
File "apache/aurora/executor/common/health_checker.py", line 248, in
from_assigned_task
pw_entry = pwd.getpwnam(assigned_task.task.job.role)
KeyError: 'getpwnam(): name not found: meta-umonitor-worker'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AURORA-1661) Scheduler leader failed to re-announce itself after ZK name was changed
Dmitriy Shirchenko created AURORA-1661:
--
Summary: Scheduler leader failed to re-announce itself after ZK
name was changed
Key: AURORA-1661
URL: https://issues.apache.org/jira/browse/AURORA-1661
Project: Aurora
Issue Type: Story
Reporter: Dmitriy Shirchenko
Assignee: John Sirois
After we renamed our ZK cluster we were in a situation where the Aurora leader
did not write to the ZK endpoint.
Sequence that we performed:
- change aurora cluster name in clusters.json
- restart all schedulers (all good)
- restart zk and move replication log (lose all state) during renaming:
intentional
- current Aurora leader fails to re-announce itself but election isn't
retriggered so we lose our leader
- some time later: restart all schedulers, and a new leader is elected (good
again)
Partial logs from the master (sorry, I'm not sure which parts are super
relevant).
{code}
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,723:30181(0x7f7586244700):ZOO_ERROR@handle_socket_error_msg@1697:
Socket [10.162.18.25:2181] zk retcode=-4, errno=111(Connection refused): server
refused to accept the client
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,723:30181(0x7f7586a45700):ZOO_ERROR@handle_socket_error_msg@1697:
Socket [10.162.28.31:2181] zk retcode=-4, errno=111(Connection refused): server
refused to accept the client
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,723:30181(0x7f7586244700):ZOO_INFO@check_events@1703: initiated
connection to server [10.162.22.24:2181]
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,723:30181(0x7f7586a45700):ZOO_INFO@check_events@1703: initiated
connection to server [10.162.2.27:2181]
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,733:30181(0x7f7586a45700):ZOO_INFO@check_events@1750: session
establishment complete on server [10.162.2.27:2181],
sessionId=0x28004d50ccb80001, negotiated timeout=4000
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.734082
30370 group.cpp:349] Group process (group(1)@10.162.12.31:8083) connected to
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.734127
30370 group.cpp:831] Syncing group operations: queue size (joins, cancels,
datas) = (0, 0, 0)
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.734136
30370 group.cpp:427] Trying to create path '/aurora/replicated-log' in ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.755173
30373 network.hpp:413] ZooKeeper group memberships changed
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.755249
30367 group.cpp:700] Trying to get '/aurora/replicated-log/00' in
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.756778
30364 network.hpp:461] ZooKeeper group PIDs: { log-replica(1)@10.162.14.30:8083
}
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.763407
30373 network.hpp:413] ZooKeeper group memberships changed
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.763478
30362 group.cpp:700] Trying to get '/aurora/replicated-log/00' in
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.764437
30362 group.cpp:700] Trying to get '/aurora/replicated-log/01' in
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.765494
30376 network.hpp:461] ZooKeeper group PIDs: {
log-replica(1)@10.160.41.62:8083, log-replica(1)@10.162.14.30:8083 }
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: 2016-04-08
22:59:10,782:30181(0x7f7586244700):ZOO_INFO@check_events@1750: session
establishment complete on server [10.162.22.24:2181],
sessionId=0x26004d8049520002, negotiated timeout=4000
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.782429
30376 group.cpp:349] Group process (group(2)@10.162.12.31:8083) connected to
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.782444
30376 group.cpp:831] Syncing group operations: queue size (joins, cancels,
datas) = (1, 0, 0)
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.782447
30376 group.cpp:427] Trying to create path '/aurora/replicated-log' in ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.795754
30377 network.hpp:413] ZooKeeper group memberships changed
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.795805
30359 group.cpp:700] Trying to get '/aurora/replicated-log/00' in
ZooKeeper
Apr 08 22:59:10 compute33-sjc1 aurora-scheduler[30176]: I0408 22:59:10.796530
30359 group.cpp:700] Trying to get '/aurora/replicated-log/01' in
ZooKeeper
Apr 08 22:59:10
[jira] [Commented] (AURORA-1641) Shell health checker is running as root
[
https://issues.apache.org/jira/browse/AURORA-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196520#comment-15196520
]
Dmitriy Shirchenko commented on AURORA-1641:
I would love to help and feel responsible but I'm going on vacation on Sunday
for a week so don't have time right now :/.
But in the meanwhile can someone give a rough outline of required work?
One proposal I saw was by [~zmanji] who mentioned that we may need to make the
health check runner look more like:
https://github.com/apache/aurora/blame/d752d466c550118f052d23519d071eb41b2e5bf6/src/main/python/apache/thermos/core/process.py#L327
> Shell health checker is running as root
> ---
>
> Key: AURORA-1641
> URL: https://issues.apache.org/jira/browse/AURORA-1641
> Project: Aurora
> Issue Type: Bug
> Components: Executor, Security
>Reporter: Stephan Erb
>Priority: Blocker
>
> As the operator of an Aurora cluster, I have to guarantee that users can run
> commands only with the privileges of their {{role}}. The new health checker
> feature is risky in that regard, as it runs all health check commands with
> the privileges of the Thermos runner. In most common deployments this is root.
> The Thermos runner supports various means for setting the uid/user/role that
> is used to run user processes. The same configuration should also apply to
> the user-defined health checking command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1622) Add ability to pass in environment variables into shell health checker
[ https://issues.apache.org/jira/browse/AURORA-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160081#comment-15160081 ] Dmitriy Shirchenko commented on AURORA-1622: On second thought, maybe just passing in the assigned port via an environment variable would do it. Just need to switch check_call to Popen here [1] and pass the port from here [2] [1] https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/common/health_check/shell.py#L50 [2] https://github.com/apache/aurora/blob/master/src/main/python/apache/aurora/executor/common/health_checker.py#L224 > Add ability to pass in environment variables into shell health checker > -- > > Key: AURORA-1622 > URL: https://issues.apache.org/jira/browse/AURORA-1622 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > > A use case is to execute the health check command based on the port > assignment. > One of the options is to change subprocess32 to use Popen funciton instead of > current check_call as Popen has an env= argument. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1622) Add ability to pass in environment variables into shell health checker
[ https://issues.apache.org/jira/browse/AURORA-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1622: --- Description: A use case is to execute the health check command based on the port assignment. One of the options is to change subprocess32 to use Popen funciton instead of current check_call as Popen has an env= argument. was:A use case is to execute the health check command based on the port assignment. > Add ability to pass in environment variables into shell health checker > -- > > Key: AURORA-1622 > URL: https://issues.apache.org/jira/browse/AURORA-1622 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko > > A use case is to execute the health check command based on the port > assignment. > One of the options is to change subprocess32 to use Popen funciton instead of > current check_call as Popen has an env= argument. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1622) Add ability to pass in environment variables into shell health checker
Dmitriy Shirchenko created AURORA-1622: -- Summary: Add ability to pass in environment variables into shell health checker Key: AURORA-1622 URL: https://issues.apache.org/jira/browse/AURORA-1622 Project: Aurora Issue Type: Task Reporter: Dmitriy Shirchenko A use case is to execute the health check command based on the port assignment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1589) Dupe debs are showing up
[ https://issues.apache.org/jira/browse/AURORA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111759#comment-15111759 ] Dmitriy Shirchenko commented on AURORA-1589: cc [~jsirois] > Dupe debs are showing up > > > Key: AURORA-1589 > URL: https://issues.apache.org/jira/browse/AURORA-1589 > Project: Aurora > Issue Type: Story >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner > > Looks like jessie and trusty packages are colliding and trusty (last one to > build) is winning out: > https://builds.apache.org/job/aurora-packaging-nightly/lastSuccessfulBuild/artifact/aurora-packaging/artifacts/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1589) Dupe debs are showing up
Dmitriy Shirchenko created AURORA-1589: -- Summary: Dupe debs are showing up Key: AURORA-1589 URL: https://issues.apache.org/jira/browse/AURORA-1589 Project: Aurora Issue Type: Story Reporter: Dmitriy Shirchenko Assignee: Bill Farner Looks like jessie and trusty packages are colliding and trusty (last one to build) is winning out: https://builds.apache.org/job/aurora-packaging-nightly/lastSuccessfulBuild/artifact/aurora-packaging/artifacts/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1562) Create separate HttpEndpointConfig and ShellEndpointConfig structs
[ https://issues.apache.org/jira/browse/AURORA-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059244#comment-15059244 ] Dmitriy Shirchenko commented on AURORA-1562: Review: https://reviews.apache.org/r/41428/ > Create separate HttpEndpointConfig and ShellEndpointConfig structs > -- > > Key: AURORA-1562 > URL: https://issues.apache.org/jira/browse/AURORA-1562 > Project: Aurora > Issue Type: Story >Affects Versions: 0.11.0 >Reporter: Dmitriy Shirchenko > > Need to address feedback from https://reviews.apache.org/r/41154/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1563) Deprecate endpoint, expected_response and expected_response_code from HealthCheckConfig
Dmitriy Shirchenko created AURORA-1563: -- Summary: Deprecate endpoint, expected_response and expected_response_code from HealthCheckConfig Key: AURORA-1563 URL: https://issues.apache.org/jira/browse/AURORA-1563 Project: Aurora Issue Type: Story Reporter: Dmitriy Shirchenko Fix For: 0.12.0 For example, remove deprecated code from health_checker.py and config.py which supports 2 ways of getting attributes listed in the title of this task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1562) Create separate HttpEndpointConfig and ShellEndpointConfig structs
[ https://issues.apache.org/jira/browse/AURORA-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1562: --- Description: Need to address feedback from https://reviews.apache.org/r/41154/ > Create separate HttpEndpointConfig and ShellEndpointConfig structs > -- > > Key: AURORA-1562 > URL: https://issues.apache.org/jira/browse/AURORA-1562 > Project: Aurora > Issue Type: Story >Affects Versions: 0.11.0 >Reporter: Dmitriy Shirchenko > > Need to address feedback from https://reviews.apache.org/r/41154/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1562) Create separate HttpEndpointConfig and ShellEndpointConfig structs
Dmitriy Shirchenko created AURORA-1562: -- Summary: Create separate HttpEndpointConfig and ShellEndpointConfig structs Key: AURORA-1562 URL: https://issues.apache.org/jira/browse/AURORA-1562 Project: Aurora Issue Type: Story Affects Versions: 0.11.0 Reporter: Dmitriy Shirchenko -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1552) Perform explicit config validation in the executor when configuring health checker
Dmitriy Shirchenko created AURORA-1552: -- Summary: Perform explicit config validation in the executor when configuring health checker Key: AURORA-1552 URL: https://issues.apache.org/jira/browse/AURORA-1552 Project: Aurora Issue Type: Story Reporter: Dmitriy Shirchenko Currently there is only client side config validation, so if someone is talking directly to scheduler jobs may not fail fast and/or it is unclear why the fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1551) Add support for non-HTTP based health checks
Dmitriy Shirchenko created AURORA-1551: -- Summary: Add support for non-HTTP based health checks Key: AURORA-1551 URL: https://issues.apache.org/jira/browse/AURORA-1551 Project: Aurora Issue Type: Story Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko We would like to be able to simply define a command which will act as a health check with a non 0 return code signifying failure. Patch attached: https://reviews.apache.org/r/41154 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1551) Add support for non-HTTP based health checks
[ https://issues.apache.org/jira/browse/AURORA-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1551: --- Assignee: Bill Farner (was: Dmitriy Shirchenko) > Add support for non-HTTP based health checks > > > Key: AURORA-1551 > URL: https://issues.apache.org/jira/browse/AURORA-1551 > Project: Aurora > Issue Type: Story >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner > Labels: Uber > > We would like to be able to simply define a command which will act as a > health check with a non 0 return code signifying failure. > Patch attached: > https://reviews.apache.org/r/41154 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1536) Add test coverage to python/src/apache/*
[ https://issues.apache.org/jira/browse/AURORA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1536: --- Assignee: Bill Farner (was: Zameer Manji) > Add test coverage to python/src/apache/* > > > Key: AURORA-1536 > URL: https://issues.apache.org/jira/browse/AURORA-1536 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Trivial > Labels: Uber > > https://reviews.apache.org/r/39958/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1536) Add test coverage to python/src/apache/*
[ https://issues.apache.org/jira/browse/AURORA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998277#comment-14998277 ] Dmitriy Shirchenko commented on AURORA-1536: Reassigned to Bill so this can be closed properly. I'm not sure how :/ > Add test coverage to python/src/apache/* > > > Key: AURORA-1536 > URL: https://issues.apache.org/jira/browse/AURORA-1536 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Trivial > Labels: Uber > > https://reviews.apache.org/r/39958/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1536) Add test coverage to python/src/apache/*
[ https://issues.apache.org/jira/browse/AURORA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1536: --- Assignee: Zameer Manji (was: Bill Farner) > Add test coverage to python/src/apache/* > > > Key: AURORA-1536 > URL: https://issues.apache.org/jira/browse/AURORA-1536 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Zameer Manji >Priority: Trivial > Labels: Uber > > https://reviews.apache.org/r/39958/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1536) Add test coverage to python/src/apache/*
[ https://issues.apache.org/jira/browse/AURORA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1536: --- Summary: Add test coverage to python/src/apache/* (was: Add test coverage python/src/apache/*) > Add test coverage to python/src/apache/* > > > Key: AURORA-1536 > URL: https://issues.apache.org/jira/browse/AURORA-1536 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Trivial > Labels: Uber > > https://reviews.apache.org/r/39958/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1531) upgrade psutil to 3.2.2 from 2.1.3
[ https://issues.apache.org/jira/browse/AURORA-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1531: --- Labels: Uber (was: uber) > upgrade psutil to 3.2.2 from 2.1.3 > -- > > Key: AURORA-1531 > URL: https://issues.apache.org/jira/browse/AURORA-1531 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Minor > Labels: Uber > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1531) upgrade psutil to 3.2.2 from 2.1.3
[ https://issues.apache.org/jira/browse/AURORA-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1531: --- Labels: uber (was: ) > upgrade psutil to 3.2.2 from 2.1.3 > -- > > Key: AURORA-1531 > URL: https://issues.apache.org/jira/browse/AURORA-1531 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Minor > Labels: uber > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AURORA-1529) Wrong command to `Build a client executable`
[ https://issues.apache.org/jira/browse/AURORA-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitriy Shirchenko updated AURORA-1529: --- Labels: Uber (was: uber) > Wrong command to `Build a client executable` > > > Key: AURORA-1529 > URL: https://issues.apache.org/jira/browse/AURORA-1529 > Project: Aurora > Issue Type: Bug >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko >Priority: Trivial > Labels: Uber > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1531) upgrade psutil to 3.2.2 from 2.1.3
[ https://issues.apache.org/jira/browse/AURORA-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983006#comment-14983006 ] Dmitriy Shirchenko commented on AURORA-1531: https://reviews.apache.org/r/39822/ > upgrade psutil to 3.2.2 from 2.1.3 > -- > > Key: AURORA-1531 > URL: https://issues.apache.org/jira/browse/AURORA-1531 > Project: Aurora > Issue Type: Task >Reporter: Dmitriy Shirchenko >Assignee: Dmitriy Shirchenko >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1530) client tests fails when run via Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
Dmitriy Shirchenko created AURORA-1530:
--
Summary: client tests fails when run via Test client code: ./pants
test src/test/python/apache/aurora/client/cli:all
Key: AURORA-1530
URL: https://issues.apache.org/jira/browse/AURORA-1530
Project: Aurora
Issue Type: Bug
Reporter: Dmitriy Shirchenko
Priority: Minor
Docs from
https://aurora.apache.org/documentation/latest/developing-aurora-client/ state
to run this to test client.
Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
This however fails eg:
{code}
shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
[master| …3]> ./pants test src/test/python/apache/aurora/client/cli:all
INFO] Detected git repository at
/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora on branch master
13:56:43 00:00 [main]
(To run a reporting server: ./pants server)
13:56:43 00:00 [bootstrap]
13:56:43 00:00 [setup]
13:56:43 00:00 [parse]
Executing tasks in goals: bootstrap -> imports -> unpack-jars ->
deferred-sources -> gen -> resolve -> compile -> resources -> test
13:56:43 00:00 [bootstrap]
13:56:43 00:00 [bootstrap-jvm-tools]
13:56:43 00:00 [imports]
13:56:43 00:00 [ivy-imports]
13:56:43 00:00 [unpack-jars]
13:56:43 00:00 [unpack-jars]
13:56:43 00:00 [deferred-sources]
13:56:43 00:00 [deferred-sources]
13:56:43 00:00 [gen]
13:56:43 00:00 [thrift]
13:56:44 00:01 [protoc]
13:56:44 00:01 [antlr]
13:56:44 00:01 [ragel]
13:56:44 00:01 [jaxb]
13:56:44 00:01 [wire]
13:56:44 00:01 [aapt]
13:56:44 00:01 [resolve]
13:56:44 00:01 [ivy]
13:56:44 00:01 [compile]
13:56:44 00:01 [compile]
13:56:44 00:01 [jvm]
13:56:44 00:01 [jvm-compilers]
13:56:44 00:01 [resources]
13:56:44 00:01 [prepare]
13:56:44 00:01 [test]
13:56:44 00:01 [run_prep_command]
13:56:44 00:01 [test]
13:56:44 00:01 [pytest]
13:56:44 00:01 [run]
== test session starts ===
platform darwin -- Python 2.7.5, pytest-2.8.2, py-1.4.30,
pluggy-0.3.1
rootdir:
/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/test/python/apache/aurora/client/cli,
inifile:
plugins: cov-2.2.0, timeout-0.5
collected 157 items
src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
src/test/python/apache/aurora/client/cli/test_client.py ..
src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
src/test/python/apache/aurora/client/cli/test_context.py
src/test/python/apache/aurora/client/cli/test_cron.py
..
src/test/python/apache/aurora/client/cli/test_inspect.py
...
src/test/python/apache/aurora/client/cli/test_cancel_update.py ..
src/test/python/apache/aurora/client/cli/test_create.py
..
src/test/python/apache/aurora/client/cli/test_diff.py ...
src/test/python/apache/aurora/client/cli/test_kill.py
.
src/test/python/apache/aurora/client/cli/test_open.py .
src/test/python/apache/aurora/client/cli/test_restart.py
...
src/test/python/apache/aurora/client/cli/test_status.py
.
src/test/python/apache/aurora/client/cli/test_config_noun.py ...
src/test/python/apache/aurora/client/cli/test_options.py
...
src/test/python/apache/aurora/client/cli/test_plugins.py .
src/test/python/apache/aurora/client/cli/test_quota.py
.
src/test/python/apache/aurora/client/cli/test_sla.py .
src/test/python/apache/aurora/client/cli/test_task.py .
src/test/python/apache/aurora/client/cli/test_supdate.py
.
src/test/python/apache/aurora/client/cli/test_update.py
.FF...F..
src/test/python/apache/aurora/client/cli/test_version.py .
FAILURES
TestUpdateCommand.test_large_with_instances_doesnt_warn
self =
def test_large_with_instances_doesnt_warn(self):
(mock_api, mock_scheduler_proxy) =
self.create_mock_api()
mock_health_check =
self.setup_health_checks(mock_api)
mock_quota_check = self.setup_quota_check()
[jira] [Comment Edited] (AURORA-1530) client tests fails when run via Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
[
https://issues.apache.org/jira/browse/AURORA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981288#comment-14981288
]
Dmitriy Shirchenko edited comment on AURORA-1530 at 10/29/15 9:20 PM:
--
a solution suggested by [~maximk] and [~zmanji] is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
Also,
{code} ./pants test.pytest --no-fast
src/test/python/apache/aurora/client/cli:all {code} takes about 58-60s
versus
{code} ./pants test src/test/python/apache/aurora/client/cli:all {code}
3 failed, 154 passed, 1 pytest-warnings in 106.37 seconds
was (Author: shirchen):
a solution suggested by [~maximk] and [~zmanji] is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
Also,
{code} ./pants test.pytest --no-fast
src/test/python/apache/aurora/client/cli:all {code} takes about 58-60s
versus
{./pants test src/test/python/apache/aurora/client/cli:all}
3 failed, 154 passed, 1 pytest-warnings in 106.37 seconds
> client tests fails when run via Test client code: ./pants test
> src/test/python/apache/aurora/client/cli:all
> ---
>
> Key: AURORA-1530
> URL: https://issues.apache.org/jira/browse/AURORA-1530
> Project: Aurora
> Issue Type: Bug
>Reporter: Dmitriy Shirchenko
>Priority: Minor
>
> Docs from
> https://aurora.apache.org/documentation/latest/developing-aurora-client/
> state to run this to test client.
> Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
> This however fails eg:
> {code}
> shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
> [master| …3]> ./pants test src/test/python/apache/aurora/client/cli:all
> INFO] Detected git repository at
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora on branch master
> 13:56:43 00:00 [main]
>(To run a reporting server: ./pants server)
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [setup]
> 13:56:43 00:00 [parse]
>Executing tasks in goals: bootstrap -> imports -> unpack-jars
> -> deferred-sources -> gen -> resolve -> compile -> resources -> test
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [bootstrap-jvm-tools]
> 13:56:43 00:00 [imports]
> 13:56:43 00:00 [ivy-imports]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [gen]
> 13:56:43 00:00 [thrift]
> 13:56:44 00:01 [protoc]
> 13:56:44 00:01 [antlr]
> 13:56:44 00:01 [ragel]
> 13:56:44 00:01 [jaxb]
> 13:56:44 00:01 [wire]
> 13:56:44 00:01 [aapt]
> 13:56:44 00:01 [resolve]
> 13:56:44 00:01 [ivy]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [jvm]
> 13:56:44 00:01 [jvm-compilers]
> 13:56:44 00:01 [resources]
> 13:56:44 00:01 [prepare]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [run_prep_command]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [pytest]
> 13:56:44 00:01 [run]
> == test session starts ===
> platform darwin -- Python 2.7.5, pytest-2.8.2,
> py-1.4.30, pluggy-0.3.1
> rootdir:
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/test/python/apache/aurora/client/cli,
> inifile:
> plugins: cov-2.2.0, timeout-0.5
> collected 157 items
>
>
> src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
> src/test/python/apache/aurora/client/cli/test_client.py
> ..
>
> src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
> src/test/python/apache/aurora/client/cli/test_context.py
>
> src/test/python/apache/aurora/client/cli/test_cron.py
> ..
> src/test/python/apache/aurora/client/cli/test_inspect.py
> ...
>
> src/test/python/apache/aurora/client/cli/test_cancel_update.py ..
> src/test/python/apache/aurora/client/cli/test_create.py
> ..
> src/test/python/apache/aurora/client/cli/test_diff.py ...
> src/test/python/apache/aurora/client/cli/test_kill.py
> .
> src/test/python/apache/aurora/client/cli/test_open.py
> .
> src/test/python/apache/aurora/client/cli/test_restart.py
> ...
>
[jira] [Comment Edited] (AURORA-1530) client tests fails when run via Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
[
https://issues.apache.org/jira/browse/AURORA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981288#comment-14981288
]
Dmitriy Shirchenko edited comment on AURORA-1530 at 10/29/15 9:20 PM:
--
a solution suggested by [~maximk] and [~zmanji] is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
Also,
{code} ./pants test.pytest --no-fast
src/test/python/apache/aurora/client/cli:all {code} takes about 58-60s
versus
{./pants test src/test/python/apache/aurora/client/cli:all}
3 failed, 154 passed, 1 pytest-warnings in 106.37 seconds
was (Author: shirchen):
a solution suggested by [~maximk]and [~zmanji] is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
> client tests fails when run via Test client code: ./pants test
> src/test/python/apache/aurora/client/cli:all
> ---
>
> Key: AURORA-1530
> URL: https://issues.apache.org/jira/browse/AURORA-1530
> Project: Aurora
> Issue Type: Bug
>Reporter: Dmitriy Shirchenko
>Priority: Minor
>
> Docs from
> https://aurora.apache.org/documentation/latest/developing-aurora-client/
> state to run this to test client.
> Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
> This however fails eg:
> {code}
> shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
> [master| …3]> ./pants test src/test/python/apache/aurora/client/cli:all
> INFO] Detected git repository at
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora on branch master
> 13:56:43 00:00 [main]
>(To run a reporting server: ./pants server)
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [setup]
> 13:56:43 00:00 [parse]
>Executing tasks in goals: bootstrap -> imports -> unpack-jars
> -> deferred-sources -> gen -> resolve -> compile -> resources -> test
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [bootstrap-jvm-tools]
> 13:56:43 00:00 [imports]
> 13:56:43 00:00 [ivy-imports]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [gen]
> 13:56:43 00:00 [thrift]
> 13:56:44 00:01 [protoc]
> 13:56:44 00:01 [antlr]
> 13:56:44 00:01 [ragel]
> 13:56:44 00:01 [jaxb]
> 13:56:44 00:01 [wire]
> 13:56:44 00:01 [aapt]
> 13:56:44 00:01 [resolve]
> 13:56:44 00:01 [ivy]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [jvm]
> 13:56:44 00:01 [jvm-compilers]
> 13:56:44 00:01 [resources]
> 13:56:44 00:01 [prepare]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [run_prep_command]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [pytest]
> 13:56:44 00:01 [run]
> == test session starts ===
> platform darwin -- Python 2.7.5, pytest-2.8.2,
> py-1.4.30, pluggy-0.3.1
> rootdir:
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/test/python/apache/aurora/client/cli,
> inifile:
> plugins: cov-2.2.0, timeout-0.5
> collected 157 items
>
>
> src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
> src/test/python/apache/aurora/client/cli/test_client.py
> ..
>
> src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
> src/test/python/apache/aurora/client/cli/test_context.py
>
> src/test/python/apache/aurora/client/cli/test_cron.py
> ..
> src/test/python/apache/aurora/client/cli/test_inspect.py
> ...
>
> src/test/python/apache/aurora/client/cli/test_cancel_update.py ..
> src/test/python/apache/aurora/client/cli/test_create.py
> ..
> src/test/python/apache/aurora/client/cli/test_diff.py ...
> src/test/python/apache/aurora/client/cli/test_kill.py
> .
> src/test/python/apache/aurora/client/cli/test_open.py
> .
> src/test/python/apache/aurora/client/cli/test_restart.py
> ...
> src/test/python/apache/aurora/client/cli/test_status.py
> .
>
> src/test/python/apache/aurora/client/cli/test_config_noun.py ...
> src/test/python/apache/aurora/client/cli/test_options.py
> ...
>
[jira] [Comment Edited] (AURORA-1530) client tests fails when run via Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
[
https://issues.apache.org/jira/browse/AURORA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981288#comment-14981288
]
Dmitriy Shirchenko edited comment on AURORA-1530 at 10/29/15 9:13 PM:
--
a solution suggested by [~maximk]and [~zmanji] is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
was (Author: shirchen):
a solution suggested by [~maximk]and [~zmanji]is to instead use
{code}
./pants test.pytest --no-fast src/test/python/apache/aurora/client/cli:all
{code}
> client tests fails when run via Test client code: ./pants test
> src/test/python/apache/aurora/client/cli:all
> ---
>
> Key: AURORA-1530
> URL: https://issues.apache.org/jira/browse/AURORA-1530
> Project: Aurora
> Issue Type: Bug
>Reporter: Dmitriy Shirchenko
>Priority: Minor
>
> Docs from
> https://aurora.apache.org/documentation/latest/developing-aurora-client/
> state to run this to test client.
> Test client code: ./pants test src/test/python/apache/aurora/client/cli:all
> This however fails eg:
> {code}
> shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
> [master| …3]> ./pants test src/test/python/apache/aurora/client/cli:all
> INFO] Detected git repository at
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora on branch master
> 13:56:43 00:00 [main]
>(To run a reporting server: ./pants server)
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [setup]
> 13:56:43 00:00 [parse]
>Executing tasks in goals: bootstrap -> imports -> unpack-jars
> -> deferred-sources -> gen -> resolve -> compile -> resources -> test
> 13:56:43 00:00 [bootstrap]
> 13:56:43 00:00 [bootstrap-jvm-tools]
> 13:56:43 00:00 [imports]
> 13:56:43 00:00 [ivy-imports]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [unpack-jars]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [deferred-sources]
> 13:56:43 00:00 [gen]
> 13:56:43 00:00 [thrift]
> 13:56:44 00:01 [protoc]
> 13:56:44 00:01 [antlr]
> 13:56:44 00:01 [ragel]
> 13:56:44 00:01 [jaxb]
> 13:56:44 00:01 [wire]
> 13:56:44 00:01 [aapt]
> 13:56:44 00:01 [resolve]
> 13:56:44 00:01 [ivy]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [compile]
> 13:56:44 00:01 [jvm]
> 13:56:44 00:01 [jvm-compilers]
> 13:56:44 00:01 [resources]
> 13:56:44 00:01 [prepare]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [run_prep_command]
> 13:56:44 00:01 [test]
> 13:56:44 00:01 [pytest]
> 13:56:44 00:01 [run]
> == test session starts ===
> platform darwin -- Python 2.7.5, pytest-2.8.2,
> py-1.4.30, pluggy-0.3.1
> rootdir:
> /Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/test/python/apache/aurora/client/cli,
> inifile:
> plugins: cov-2.2.0, timeout-0.5
> collected 157 items
>
>
> src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
> src/test/python/apache/aurora/client/cli/test_client.py
> ..
>
> src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
> src/test/python/apache/aurora/client/cli/test_context.py
>
> src/test/python/apache/aurora/client/cli/test_cron.py
> ..
> src/test/python/apache/aurora/client/cli/test_inspect.py
> ...
>
> src/test/python/apache/aurora/client/cli/test_cancel_update.py ..
> src/test/python/apache/aurora/client/cli/test_create.py
> ..
> src/test/python/apache/aurora/client/cli/test_diff.py ...
> src/test/python/apache/aurora/client/cli/test_kill.py
> .
> src/test/python/apache/aurora/client/cli/test_open.py
> .
> src/test/python/apache/aurora/client/cli/test_restart.py
> ...
> src/test/python/apache/aurora/client/cli/test_status.py
> .
>
> src/test/python/apache/aurora/client/cli/test_config_noun.py ...
> src/test/python/apache/aurora/client/cli/test_options.py
> ...
> src/test/python/apache/aurora/client/cli/test_plugins.py
> .
> src/test/python/apache/aurora/client/cli/test_quota.py
> .
> src/test/python/apache/aurora/client/cli/test_sla.py
> .
> src/test/python/apache/aurora/client/cli/test_task.py
> .
>
[jira] [Commented] (AURORA-1529) Wrong command to `Build a client executable`
[ https://issues.apache.org/jira/browse/AURORA-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979787#comment-14979787 ] Dmitriy Shirchenko commented on AURORA-1529: oh, and i think one of the committers needs to land the diff for me. thanks! > Wrong command to `Build a client executable` > > > Key: AURORA-1529 > URL: https://issues.apache.org/jira/browse/AURORA-1529 > Project: Aurora > Issue Type: Bug >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1529) Wrong command to `Build a client executable`
[ https://issues.apache.org/jira/browse/AURORA-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979791#comment-14979791 ] Dmitriy Shirchenko commented on AURORA-1529: oh sorry, i thought i addressed it. the second command had to be changed, right? https://reviews.apache.org/r/39742/diff/3#index_header > Wrong command to `Build a client executable` > > > Key: AURORA-1529 > URL: https://issues.apache.org/jira/browse/AURORA-1529 > Project: Aurora > Issue Type: Bug >Reporter: Dmitriy Shirchenko >Assignee: Bill Farner >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1529) Wrong command to `Build a client executable`
Dmitriy Shirchenko created AURORA-1529: -- Summary: Wrong command to `Build a client executable` Key: AURORA-1529 URL: https://issues.apache.org/jira/browse/AURORA-1529 Project: Aurora Issue Type: Bug Reporter: Dmitriy Shirchenko Assignee: Dmitriy Shirchenko Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1529) Wrong command to `Build a client executable`
[
https://issues.apache.org/jira/browse/AURORA-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979487#comment-14979487
]
Dmitriy Shirchenko edited comment on AURORA-1529 at 10/28/15 11:40 PM:
---
pants docs here [1] and [2] to build a client executable :
`Build a client executable: ./pants binary
src/main/python/apache/aurora/client/cli:aurora`
`./pants binary src/main/python/apache/aurora/client/cli:kaurora`
leads to
`
{code}
(env)
shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
[master| …443]> ./pants binary src/main/python/apache/aurora/client/cli:aurora
Exception caught:
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/bin/pants",
line 9, in
load_entry_point('pantsbuild.pants==0.0.32', 'console_scripts', 'pants')()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py",
line 81, in main
_run(exiter)
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py",
line 72, in _run
goal_runner.setup()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py",
line 109, in setup
self._expand_goals_and_specs()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py",
line 163, in _expand_goals_and_specs
for address in spec_parser.parse_addresses(spec, fail_fast):
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/base/cmd_line_spec_parser.py",
line 85, in parse_addresses
for address in self._parse_spec(spec, fail_fast):
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/base/cmd_line_spec_parser.py",
line 172, in _parse_spec
raise self.BadSpecError(e)
Exception message: BUILD file does not exist at:
/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/main/python/apache/aurora/client/cli/BUILD
{code}
Similar story for the kerberos client.
[1] http://aurora.apache.org/documentation/latest/developing-aurora-client
[2] http://aurora.apache.org/documentation/latest/security/
was (Author: shirchen):
pants docs here [1] and [2] to build a client executable :
`Build a client executable: ./pants binary
src/main/python/apache/aurora/client/cli:aurora`
`./pants binary src/main/python/apache/aurora/client/cli:kaurora`
leads to
`
(env)
shirchen@Dmitriys-MacBook-Pro-7:~/Uber/sync/glitz.dev.uber.com/home/uber/aurora
[master| …443]> ./pants binary src/main/python/apache/aurora/client/cli:aurora
Exception caught:
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/bin/pants",
line 9, in
load_entry_point('pantsbuild.pants==0.0.32', 'console_scripts', 'pants')()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py",
line 81, in main
_run(exiter)
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/pants_exe.py",
line 72, in _run
goal_runner.setup()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py",
line 109, in setup
self._expand_goals_and_specs()
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/bin/goal_runner.py",
line 163, in _expand_goals_and_specs
for address in spec_parser.parse_addresses(spec, fail_fast):
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/base/cmd_line_spec_parser.py",
line 85, in parse_addresses
for address in self._parse_spec(spec, fail_fast):
File
"/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/build-support/pants.venv/lib/python2.7/site-packages/pants/base/cmd_line_spec_parser.py",
line 172, in _parse_spec
raise self.BadSpecError(e)
Exception message: BUILD file does not exist at:
/Users/shirchen/Uber/sync/glitz.dev.uber.com/home/uber/aurora/src/main/python/apache/aurora/client/cli/BUILD
`
[1] http://aurora.apache.org/documentation/latest/developing-aurora-client
[2] http://aurora.apache.org/documentation/latest/security/
> Wrong command to `Build a client executable`
>
>
> Key: AURORA-1529
> URL:
