[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421854#comment-15421854
]
Santhosh Kumar Shanmugham commented on AURORA-1711:
---
+1 on extending {{StartJobUpdateResult}}
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421467#comment-15421467
]
Joshua Cohen commented on AURORA-1711:
--
Ok, that sounds reasonable enough to me. Especially if we extend the
{{StartJobUpdateResult}} to make it possible to avoid the query entirely.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421444#comment-15421444
]
Maxim Khutornenko commented on AURORA-1711:
---
I think saving an extra read-only RPC call in case of a failed
{{startJobUpdate}} call is a questionable tradeoff here. The rate of
{{startJobUpdate}} calls is incomparable to UI and other aurora client
auto-retry pulls we already have in the system.
We can further improve and save an extra call by extending the
{{StartJobUpdateResult}} to have an optional metadata or the entire
{{JobUpdate}} object with the in-progress update details. That said, I feel
relying on an occasional {{getJobUpdateDetails}} call placed *only* in case of
a failed {{startJobUpdate}} should be sufficient here.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421363#comment-15421363
]
Joshua Cohen commented on AURORA-1711:
--
My concern is that if we want to support this in the client (and I feel
strongly that we should), then we're always going to make this query when
starting a job update (or at the very least, we'll always make this query when
retrying a failed {{startJobUpdate}} call). Given that, it seems beneficial to
me to incorporate this directly into the scheduler. Imagine the scenario where
the scheduler is failing the {{startJobUpdate}} request for reasons other than
a network partition, e.g. for whatever reason it's overloaded. If the client
always responds to that failure by first querying to see if the update it's
trying to launch is in progress and *then* retries the {{startJobUpdate}} call
that's only going to exacerbate the underlying load on the scheduler. Whereas
if the scheduler performs this check implicitly if the client-update-id is set
on the {{JobUpdateRequest}} the process can be better optimized and avoid the
overhead of an unnecessary RPC.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421283#comment-15421283
]
Maxim Khutornenko commented on AURORA-1711:
---
{quote}
In that scenario, the client will essentially always make the query to see if
the update is active
{quote}
I don't think optimizing this hopefully rare case of network partitioning is
enough justification for the scheduler to take act upon an obscure
user-controlled field. Having action separated from data opens up a much richer
set of use cases for the job update metadata and does not bind scheduler into
contract with the client.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421267#comment-15421267
]
Joshua Cohen commented on AURORA-1711:
--
Does forcing the client to make the query become unnecessarily expensive? The
aurora command line client can also benefit from this logic in the scenario
where you're accessing the API through a proxy (e.g. the {{startUpdateRequest}}
cal is sent through the proxy to the scheduler, the update begins, but for
whatever reason the proxy times out the request to the client, at which point
the client automatically retries the request and gets an error about the update
already being in progress). In that scenario, the client will essentially
*always* make the query to see if the update is active, so why not just have
the scheduler implicitly make the check when processing a {{startJobUpdate}}
request that includes the metadata? I think it would also makes sense, in this
case, to move away from a generic "metadata" field and towards an explicit
"client update identifier" field so that the scheduler is not enforcing its own
meaning on what from the outside appears to be data whose purpose should be
unknown to the scheduler.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421267#comment-15421267
]
Joshua Cohen edited comment on AURORA-1711 at 8/15/16 4:59 PM:
---
Does forcing the client to make the query become unnecessarily expensive? The
aurora command line client can also benefit from this logic in the scenario
where you're accessing the API through a proxy (e.g. the {{startJobUpdate}} cal
is sent through the proxy to the scheduler, the update begins, but for whatever
reason the proxy times out the request to the client, at which point the client
automatically retries the request and gets an error about the update already
being in progress). In that scenario, the client will essentially *always* make
the query to see if the update is active, so why not just have the scheduler
implicitly make the check when processing a {{startJobUpdate}} request that
includes the metadata? I think it would also makes sense, in this case, to move
away from a generic "metadata" field and towards an explicit "client update
identifier" field so that the scheduler is not enforcing its own meaning on
what from the outside appears to be data whose purpose should be unknown to the
scheduler.
was (Author: joshua.cohen):
Does forcing the client to make the query become unnecessarily expensive? The
aurora command line client can also benefit from this logic in the scenario
where you're accessing the API through a proxy (e.g. the {{startUpdateRequest}}
cal is sent through the proxy to the scheduler, the update begins, but for
whatever reason the proxy times out the request to the client, at which point
the client automatically retries the request and gets an error about the update
already being in progress). In that scenario, the client will essentially
*always* make the query to see if the update is active, so why not just have
the scheduler implicitly make the check when processing a {{startJobUpdate}}
request that includes the metadata? I think it would also makes sense, in this
case, to move away from a generic "metadata" field and towards an explicit
"client update identifier" field so that the scheduler is not enforcing its own
meaning on what from the outside appears to be data whose purpose should be
unknown to the scheduler.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1711) Allow client to store metadata on Update entity
[ https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421240#comment-15421240 ] Maxim Khutornenko commented on AURORA-1711: --- I like the idea of the update metadata but don't like the part where scheduler would be required to look into it. The metadata field should be a free-form (as far as scheduler goes) string or key/value data storing any arbitrary values provided by the client. Clients are free to interpret that data in the desired way, including querying for active updates to enforce the uniqueness requirements. > Allow client to store metadata on Update entity > --- > > Key: AURORA-1711 > URL: https://issues.apache.org/jira/browse/AURORA-1711 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: David McLaughlin > > I have a use case where I'm programmatically starting updates via the Aurora > API and sometimes the request to the scheduler times out or fails, even > though the update is written to storage and started. > I'd like to be able to store some unique identifier on the update so that we > can reconcile this state later. We can make this generic by allowing clients > to store arbitrary metadata on an update (similar to how they do it with job > configuration). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
