[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421267#comment-15421267
]
Joshua Cohen edited comment on AURORA-1711 at 8/15/16 4:59 PM:
---
Does forcing the client to make the query become unnecessarily expensive? The
aurora command line client can also benefit from this logic in the scenario
where you're accessing the API through a proxy (e.g. the {{startJobUpdate}} cal
is sent through the proxy to the scheduler, the update begins, but for whatever
reason the proxy times out the request to the client, at which point the client
automatically retries the request and gets an error about the update already
being in progress). In that scenario, the client will essentially *always* make
the query to see if the update is active, so why not just have the scheduler
implicitly make the check when processing a {{startJobUpdate}} request that
includes the metadata? I think it would also makes sense, in this case, to move
away from a generic "metadata" field and towards an explicit "client update
identifier" field so that the scheduler is not enforcing its own meaning on
what from the outside appears to be data whose purpose should be unknown to the
scheduler.
was (Author: joshua.cohen):
Does forcing the client to make the query become unnecessarily expensive? The
aurora command line client can also benefit from this logic in the scenario
where you're accessing the API through a proxy (e.g. the {{startUpdateRequest}}
cal is sent through the proxy to the scheduler, the update begins, but for
whatever reason the proxy times out the request to the client, at which point
the client automatically retries the request and gets an error about the update
already being in progress). In that scenario, the client will essentially
*always* make the query to see if the update is active, so why not just have
the scheduler implicitly make the check when processing a {{startJobUpdate}}
request that includes the metadata? I think it would also makes sense, in this
case, to move away from a generic "metadata" field and towards an explicit
"client update identifier" field so that the scheduler is not enforcing its own
meaning on what from the outside appears to be data whose purpose should be
unknown to the scheduler.
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419776#comment-15419776
]
Santhosh Kumar Shanmugham edited comment on AURORA-1711 at 8/15/16 4:53 PM:
Incorporating [~maximk] and [~davmclau]'s suggestions to put forward a proposal.
Client will send the UUID as part of the metadata field in {{JobUpdateRequest}}
like,
{code}
JobUpdateRequest {
TaskConfig {
JobKey : “role/env/name”
Metadata : {
“Disambiguator” : “FGFJAGGFIAHKHGFAK” #UUID
}
}
}
{code}
Scheduler returns the active updates for the {{JobKey}} along with the
{{Disambiguator}} to the Client, who can reconcile the requests.
*Pros:*
- Logic is centralized and can benefit all the clients
- No costly diffs are needed
- No explicit API change
*Cons:*
- Scheduler and Client will need changes
- Possibility of identifier collision
- Additional query to the store
https://docs.google.com/a/twitter.com/document/d/1Ih-WXACZiPB0Z8EAQw_8cnAaf4eHMsUGG3B3OKia0k4/edit?usp=sharing
was (Author: santhk):
Incorporating [~maximk] and [~davmclau]'s suggestions to put forward a proposal.
Client will send the UUID as part of the metadata field in {{JobUpdateRequest}}
like,
{code}
JobUpdateRequest {
TaskConfig {
JobKey : “role/env/name”
Metadata : {
“Disambiguator” : “FGFJAGGFIAHKHGFAK” #UUID
}
}
}
{code}
Scheduler looks up the active updates for the {{JobKey}} and compare the
{{Disambiguator}} to make sure the request is not a duplicate.
*Pros:*
- Logic is centralized and can benefit all the clients
- No costly diffs are needed
- No explicit API change
*Cons:*
- Scheduler and Client will need changes
- Possibility of identifier collision
- Additional query to the store
https://docs.google.com/a/twitter.com/document/d/1Ih-WXACZiPB0Z8EAQw_8cnAaf4eHMsUGG3B3OKia0k4/edit?usp=sharing
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[
https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419776#comment-15419776
]
Santhosh Kumar Shanmugham edited comment on AURORA-1711 at 8/13/16 1:46 AM:
Incorporating [~maximk] and [~davmclau]'s suggestions to put forward a proposal.
Client will send the UUID as part of the metadata field in {{JobUpdateRequest}}
like,
{code}
JobUpdateRequest {
TaskConfig {
JobKey : “role/env/name”
Metadata : {
“Disambiguator” : “FGFJAGGFIAHKHGFAK” #UUID
}
}
}
{code}
Scheduler looks up the active updates for the {{JobKey}} and compare the
{{Disambiguator}} to make sure the request is not a duplicate.
*Pros:*
- Logic is centralized and can benefit all the clients
- No costly diffs are needed
- No explicit API change
*Cons:*
- Scheduler and Client will need changes
- Possibility of identifier collision
- Additional query to the store
https://docs.google.com/a/twitter.com/document/d/1Ih-WXACZiPB0Z8EAQw_8cnAaf4eHMsUGG3B3OKia0k4/edit?usp=sharing
was (Author: santhk):
Incorporating [~maximk] and [~davmclau]'s suggestions to put forward a proposal.
Client will send the UUID as part of the metadata field in JobUpdateRequest
like,
{code}
JobUpdateRequest {
TaskConfig {
JobKey : “role/env/name”
Metadata : {
“Disambiguator” : “FGFJAGGFIAHKHGFAK” #UUID
}
}
}
{code}
Scheduler looks up the active updates for the {{JobKey}} and compare the
{{Disambiguator}} to make sure the request is not a duplicate.
*Pros:*
- Logic is centralized and can benefit all the clients
- No costly diffs are needed
- No explicit API change
*Cons:*
- Scheduler and Client will need changes
- Possibility of identifier collision
- Additional query to the store
https://docs.google.com/a/twitter.com/document/d/1Ih-WXACZiPB0Z8EAQw_8cnAaf4eHMsUGG3B3OKia0k4/edit?usp=sharing
> Allow client to store metadata on Update entity
> ---
>
> Key: AURORA-1711
> URL: https://issues.apache.org/jira/browse/AURORA-1711
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: David McLaughlin
>
> I have a use case where I'm programmatically starting updates via the Aurora
> API and sometimes the request to the scheduler times out or fails, even
> though the update is written to storage and started.
> I'd like to be able to store some unique identifier on the update so that we
> can reconcile this state later. We can make this generic by allowing clients
> to store arbitrary metadata on an update (similar to how they do it with job
> configuration).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[ https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335233#comment-15335233 ] David McLaughlin edited comment on AURORA-1711 at 6/17/16 2:23 AM: --- In our case it's not about correctness (our tool can continue because we eventually detect a no-op update), but it's about auditing and tracking. We want to make sure that the action is recorded and not confuse our users. Example of user experience right now: 1) User clicks the update button for a given job key 2) The request to the Scheduler times out, so we propagate the error to the user 3) User sees a "Update failed" message in the UI/API 4) User tries again, gets a "Update in progress" message. 5) User is confused for the duration of their update. 6) Eventually the update completes and our system lets them skip over the 'no-op' update. 7) Another user comes along and wonders why a given release was skipped. Also if you replace the user with an automated orchestration agent, it becomes even more important to be able to reconcile the state. was (Author: davmclau): In our case it's not about correctness (our tool can continue because eventually we eventually detect a no-op update), but it's about auditing and tracking. We want to make sure that the action is recorded and remove confusion to users. Example of user experience right now: 1) User clicks the update button for a given job key 2) The request to the Scheduler times out, so we propagate the error to the user 3) User sees a "Update failed" message in the UI/API 4) User tries again, gets a "Update in progress" message. 5) User is confused for the duration of their update. 6) Eventually the update completes and our system lets them skip over the 'no-op' update. 7) Another user comes along and wonders why a given release was skipped. Also if you replace the user with an automated orchestration agent, it becomes even more important to be able to reconcile the state. > Allow client to store metadata on Update entity > --- > > Key: AURORA-1711 > URL: https://issues.apache.org/jira/browse/AURORA-1711 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: David McLaughlin > > I have a use case where I'm programmatically starting updates via the Aurora > API and sometimes the request to the scheduler times out or fails, even > though the update is written to storage and started. > I'd like to be able to store some unique identifier on the update so that we > can reconcile this state later. We can make this generic by allowing clients > to store arbitrary metadata on an update (similar to how they do it with job > configuration). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AURORA-1711) Allow client to store metadata on Update entity
[ https://issues.apache.org/jira/browse/AURORA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335225#comment-15335225 ] Bill Farner edited comment on AURORA-1711 at 6/17/16 2:16 AM: -- What i'm not clear on is why the specific _request_ needs to be uniquely identified, since the operation is idempotent. Idempotence guarantees that multiple clients can safely step on each other, provided they are trying to achieve the same state. Does that make sense here, or am i off-base? edit: s/instances/clients/ for clarity was (Author: wfarner): What i'm not clear on is why the specific _request_ needs to be uniquely identified, since the operation is idempotent. Idempotence guarantees that multiple instances can safely step on each other, provided they are trying to achieve the same state. Does that make sense here, or am i off-base? > Allow client to store metadata on Update entity > --- > > Key: AURORA-1711 > URL: https://issues.apache.org/jira/browse/AURORA-1711 > Project: Aurora > Issue Type: Task > Components: Scheduler >Reporter: David McLaughlin > > I have a use case where I'm programmatically starting updates via the Aurora > API and sometimes the request to the scheduler times out or fails, even > though the update is written to storage and started. > I'd like to be able to store some unique identifier on the update so that we > can reconcile this state later. We can make this generic by allowing clients > to store arbitrary metadata on an update (similar to how they do it with job > configuration). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
