Re: [vdsm] [Draft]Task Management API
- Original Message -
> Dan rightly suggested I'd be more specific about what the task system
> is
> instead of what the task system isn't.
>
> The problem is that I'm not completely sure how it's going to work.
> It also depends on the events mechanism.
> This is my current working draft:
>
>
> TaskInfo:
> id string
> methodName string
> kwargs json-object (string keys variant values) *filtered to remove
> sensitive
> information
>
> getRunningTasks(filter string, filterType enum{glob, regexp})
> Returns a list of TaskInfo of all tasks that their id's match the
> filter
>
>
> That's it, not even stopTask()
So for each verb we need to implement a specific 'cancel' op if relevant?
e.g. copyImage -> cancelCopyImage ? (or just deleteImage on the target image in
this case?)
for migrateVM -> cancelMigrateVM? etc.
>
> As explained, I would like to offload handling to the subsystems.
> In order to make things easier for the clients every subsystem can
> choose a
> filed of the object to be of type OperationInfo.
> This is a generic structure that the user has a generic way to track
> all tasks
> on all subsystem with a report interface. The extraData field is for
> subsystem
> specific data. This is where the storage subsystem would put, for
> example,
> imageState (broken, degraded, optimized) data.
>
> OperationInfo:
> operationDescription string - something out of an agreed enum of
> strings
> vaguely describing the operation at
> hand for
> example "Copying", "Merging", "Deleting",
> "Configuring", "Stopped", "Paused",
> They must be known to the client so it can in
> turn translate it in the UI. The also have to
> remain relatively vague as they are part of the
> interface meaning that new values will break old
> clients so they have to be reusable.
> stageDescription - Similar to operation description in case you want
> more
>granularity, optional.
> stage (int, int) - (5, 10) means 5 out of 10. 1 out of 1 implies the
> UI to not
>display stage widgets.
> percentage - 0-100, -1 means unknown.
I'm assuming this is not overall percentage but relative to current stage?
> lastError - (code, message) the same errors that can return for
> regular calls
> extraData - json-object
>
>
> For example creatVM will return once the object is created in VDSM.
> getVmInfo() would return, amongst other things, the operation info.
> For the case of preparing for launch it will be:
> {"Creating", "configuring", (2, 4), 40, (0, ""),
>{state="preparing for launch"}}
> In the case of VM paused on EIO:
> {"Paused", "Paused", (1, 1), -1, (123, "Error writing to disks"),
>{state="paused"}}
>
> Migration is a tricky one, it will be reported as a task while it's
> in progress
> but all the information is available on the image operationInfo.
> In the case of Migration:
> {"Migration", "Configuring", (1, 3), -1, (0, ""),
> {status="Migrating"}}
Since you wrote 'image' I'm assuming you're referring to storage migration here?
So the migrate command is successful the moment it starts? (or even a bit
before, once it persisted info or something) and then to query the state of the
migration user needs to call 'image operationInfo'?
>
> For StorageConnection this is somewhat already the case but in
> simplified
> version.
>
> If you want to ask about any other operation I'd be more then happy
> to write my
> suggestion for it.
>
> Subsystems have complete freedom about how to set up the API.
> For Storage you have Fixes() to start\stop operations.
Fixes?
> Gluster is pretty autonomous once operations have been started.
>
> Since operations return as soon as they are registered (persisted) or
> fail to
> register, it makes synchronous programming a bit clunky.
> vdsm.pauseVm(vmId) doesn't return when the VM is paused but when VDSM
> committed
> it will try to pause it. This means you will have to poll in order to
> see if
> the operation finished. For gluster, as an example, this is the only
> way we
> can check that the operation finished.
>
> For stuff we have a bit more control over vdsm will fire events using
> json-rpc
> notifications sent to the clients. The will be in the form of:
> {"method": "alert", "params": {
> "alertName": (.)?..(.,
> ...),
> "operationInfo", OperationInfo}
> }
>
> The user can register to recive events using a glob or a regexp.
> registering to vdsm.VM.* pop every time any VM has changed stage.
> This means that whenever the task finishes, fails or gains
> significance progress
> and VDSM is there to track it, an event will be sent to the client.
>
> This means that the general flow is.
> # Register operation
Re: [vdsm] Managing async tasks
- Original Message - > From: "Ayal Baron" > To: "Saggi Mizrahi" > Cc: "Dan Kenigsberg" , "Federico Simoncelli" > , engine-de...@ovirt.org, > vdsm-devel@lists.fedorahosted.org, "Adam Litke" > Sent: Monday, December 17, 2012 5:24:48 PM > Subject: Re: Managing async tasks > > > > - Original Message - > > This is an addendum to my previous email. > > > > - Original Message - > > > From: "Saggi Mizrahi" > > > To: "Adam Litke" > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > , "Federico Simoncelli" > > > , engine-de...@ovirt.org, > > > vdsm-devel@lists.fedorahosted.org > > > Sent: Monday, December 17, 2012 2:52:06 PM > > > Subject: Re: Managing async tasks > > > > > > > > > > > > - Original Message - > > > > From: "Adam Litke" > > > > To: "Saggi Mizrahi" > > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > > , "Federico Simoncelli" > > > > , engine-de...@ovirt.org, > > > > vdsm-devel@lists.fedorahosted.org > > > > Sent: Monday, December 17, 2012 2:16:25 PM > > > > Subject: Re: Managing async tasks > > > > > > > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Adam Litke" To: > > > > > > vdsm-devel@lists.fedorahosted.org > > > > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > > > > , > > > > > > "Saggi Mizrahi" , "Federico > > > > > > Simoncelli" > > > > > > , engine-de...@ovirt.org Sent: Monday, > > > > > > December 17, > > > > > > 2012 12:00:49 PM Subject: Managing async tasks > > > > > > > > > > > > On today's vdsm call we had a lively discussion around how > > > > > > asynchronous > > > > > > operations should be handled in the future. In an effort > > > > > > to > > > > > > include more > > > > > > people in the discussion and to better capture the > > > > > > resulting > > > > > > conversation I > > > > > > would like to continue that discussion here on the mailing > > > > > > list. > > > > > > > > > > > > A lot of ideas were thrown around about how 'tasks' should > > > > > > be > > > > > > handled in the > > > > > > future. There are a lot of ways that it can be done. To > > > > > > determine how we > > > > > > should implement it, it's probably best if we start with a > > > > > > set > > > > > > of > > > > > > requirements. If we can first agree on these, it should be > > > > > > easy > > > > > > to find a > > > > > > solution that meets them. I'll take a stab at identifying > > > > > > a > > > > > > first set of > > > > > > POSSIBLE requirements: > > > > > > > > > > > > - Standardized method for determining the result of an > > > > > > operation > > > > > > > > > > > > This is a big one for me because it directly affects the > > > > > > consumability of > > > > > > the API. If each verb has different semantics for > > > > > > discovering > > > > > > whether it > > > > > > has completed successfully, then the API will be nearly > > > > > > impossible to use > > > > > > easily. > > > > > Since there is no way to assure if of some tasks completed > > > > > successfully or > > > > > failed, especially around the murky waters of storage, I say > > > > > this > > > > > requirement > > > > > should be removed. At least not in the context of a task. > > > > > > > > I don't agree. Please feel free to convince me with some > > > > exampled. > > > > If we > > > > cannot provide feedback to a user as to whether their request > > > > has > > > > been satisfied > > > > or not, then we have some bigger problems to solve. > > > If VDSM sends a write command to a storage server, and the > > > connection > > > hangs up before the ACK has returned. > > > The operation has been committed but VDSM has no way of knowing > > > if > > > that happened as far as VDSM is concerned it got an ETIMEO or > > > EIO. > > > This is the same problem that the engine has with VDSM. > > > If VDSM creates an image\VM\network\repo but the connection hangs > > > up > > > before the response can be sent back as far as the engine is > > > concerned the operation times out. > > > This is an inherent issue with clustering. > > > This is why I want to move away from tasks being *the* trackable > > > objects. > > > Tasks should be short. As short as possible. > > > Run VM should just persist the VM information on the VDSM host > > > and > > > return. The rest of the tracking should be done using the VM ID. > > > Create image should return once VDSM persisted the information > > > about > > > the request on the repository and created the metadata files. > > > Tracking should be done on the repo or the imageId. > > > > The thing is that I know how long a VM object should live (or an > > Image object). > > So tracking it is straight forward. How long a task should live is > > very problematic and quite context specific. > > It depends on what the task is. > > I think it's quite confusing from an API standpoint to have every > > task have a different scope, id requirement and life-cycl
Re: [vdsm] Managing async tasks
On Mon, Dec 17, 2012 at 03:12:34PM -0500, Saggi Mizrahi wrote: > This is an addendum to my previous email. > > - Original Message - > > From: "Saggi Mizrahi" > > To: "Adam Litke" > > Cc: "Dan Kenigsberg" , "Ayal Baron" , > > "Federico Simoncelli" > > , engine-de...@ovirt.org, > > vdsm-devel@lists.fedorahosted.org > > Sent: Monday, December 17, 2012 2:52:06 PM > > Subject: Re: Managing async tasks > > > > > > > > - Original Message - > > > From: "Adam Litke" > > > To: "Saggi Mizrahi" > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > , "Federico Simoncelli" > > > , engine-de...@ovirt.org, > > > vdsm-devel@lists.fedorahosted.org > > > Sent: Monday, December 17, 2012 2:16:25 PM > > > Subject: Re: Managing async tasks > > > > > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > > > > > > > > > > - Original Message - > > > > > From: "Adam Litke" To: > > > > > vdsm-devel@lists.fedorahosted.org > > > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > > > , > > > > > "Saggi Mizrahi" , "Federico Simoncelli" > > > > > , engine-de...@ovirt.org Sent: Monday, > > > > > December 17, > > > > > 2012 12:00:49 PM Subject: Managing async tasks > > > > > > > > > > On today's vdsm call we had a lively discussion around how > > > > > asynchronous > > > > > operations should be handled in the future. In an effort to > > > > > include more > > > > > people in the discussion and to better capture the resulting > > > > > conversation I > > > > > would like to continue that discussion here on the mailing > > > > > list. > > > > > > > > > > A lot of ideas were thrown around about how 'tasks' should be > > > > > handled in the > > > > > future. There are a lot of ways that it can be done. To > > > > > determine how we > > > > > should implement it, it's probably best if we start with a set > > > > > of > > > > > requirements. If we can first agree on these, it should be > > > > > easy > > > > > to find a > > > > > solution that meets them. I'll take a stab at identifying a > > > > > first set of > > > > > POSSIBLE requirements: > > > > > > > > > > - Standardized method for determining the result of an > > > > > operation > > > > > > > > > > This is a big one for me because it directly affects the > > > > > consumability of > > > > > the API. If each verb has different semantics for > > > > > discovering > > > > > whether it > > > > > has completed successfully, then the API will be nearly > > > > > impossible to use > > > > > easily. > > > > Since there is no way to assure if of some tasks completed > > > > successfully or > > > > failed, especially around the murky waters of storage, I say this > > > > requirement > > > > should be removed. At least not in the context of a task. > > > > > > I don't agree. Please feel free to convince me with some exampled. > > > If we > > > cannot provide feedback to a user as to whether their request has > > > been satisfied > > > or not, then we have some bigger problems to solve. > > If VDSM sends a write command to a storage server, and the connection > > hangs up before the ACK has returned. > > The operation has been committed but VDSM has no way of knowing if > > that happened as far as VDSM is concerned it got an ETIMEO or EIO. > > This is the same problem that the engine has with VDSM. > > If VDSM creates an image\VM\network\repo but the connection hangs up > > before the response can be sent back as far as the engine is > > concerned the operation times out. > > This is an inherent issue with clustering. > > This is why I want to move away from tasks being *the* trackable > > objects. > > Tasks should be short. As short as possible. > > Run VM should just persist the VM information on the VDSM host and > > return. The rest of the tracking should be done using the VM ID. > > Create image should return once VDSM persisted the information about > > the request on the repository and created the metadata files. > > Tracking should be done on the repo or the imageId. > > The thing is that I know how long a VM object should live (or an Image > object). > So tracking it is straight forward. How long a task should live is very > problematic and quite context specific. > It depends on what the task is. > I think it's quite confusing from an API standpoint to have every task have a > different scope, id requirement and life-cycle. > > In VDSM has two types of APIs > > CRUD objects - VM, Image, Repository, Bridge, Storage Connections > General transient methods - getBiosInfo(), getDeviceList() > > The latter are quite simple to manage. They don't need any special handling. > If you lost a getBiosInfo() call you just send another one, no harm done. > The same is even true with things that "change" the host like getDeviceList() > > What we are really arguing about is fitting the CRUD objects to some generic > task oriented scheme. > I'm saying it's a waste of time as you can quite easily have flows to recover >
[vdsm] [Draft]Task Management API
Dan rightly suggested I'd be more specific about what the task system is
instead of what the task system isn't.
The problem is that I'm not completely sure how it's going to work.
It also depends on the events mechanism.
This is my current working draft:
TaskInfo:
id string
methodName string
kwargs json-object (string keys variant values) *filtered to remove sensitive
information
getRunningTasks(filter string, filterType enum{glob, regexp})
Returns a list of TaskInfo of all tasks that their id's match the filter
That's it, not even stopTask()
As explained, I would like to offload handling to the subsystems.
In order to make things easier for the clients every subsystem can choose a
filed of the object to be of type OperationInfo.
This is a generic structure that the user has a generic way to track all tasks
on all subsystem with a report interface. The extraData field is for subsystem
specific data. This is where the storage subsystem would put, for example,
imageState (broken, degraded, optimized) data.
OperationInfo:
operationDescription string - something out of an agreed enum of strings
vaguely describing the operation at hand for
example "Copying", "Merging", "Deleting",
"Configuring", "Stopped", "Paused",
They must be known to the client so it can in
turn translate it in the UI. The also have to
remain relatively vague as they are part of the
interface meaning that new values will break old
clients so they have to be reusable.
stageDescription - Similar to operation description in case you want more
granularity, optional.
stage (int, int) - (5, 10) means 5 out of 10. 1 out of 1 implies the UI to not
display stage widgets.
percentage - 0-100, -1 means unknown.
lastError - (code, message) the same errors that can return for regular calls
extraData - json-object
For example creatVM will return once the object is created in VDSM.
getVmInfo() would return, amongst other things, the operation info.
For the case of preparing for launch it will be:
{"Creating", "configuring", (2, 4), 40, (0, ""),
{state="preparing for launch"}}
In the case of VM paused on EIO:
{"Paused", "Paused", (1, 1), -1, (123, "Error writing to disks"),
{state="paused"}}
Migration is a tricky one, it will be reported as a task while it's in progress
but all the information is available on the image operationInfo.
In the case of Migration:
{"Migration", "Configuring", (1, 3), -1, (0, ""), {status="Migrating"}}
For StorageConnection this is somewhat already the case but in simplified
version.
If you want to ask about any other operation I'd be more then happy to write my
suggestion for it.
Subsystems have complete freedom about how to set up the API.
For Storage you have Fixes() to start\stop operations.
Gluster is pretty autonomous once operations have been started.
Since operations return as soon as they are registered (persisted) or fail to
register, it makes synchronous programming a bit clunky.
vdsm.pauseVm(vmId) doesn't return when the VM is paused but when VDSM committed
it will try to pause it. This means you will have to poll in order to see if
the operation finished. For gluster, as an example, this is the only way we
can check that the operation finished.
For stuff we have a bit more control over vdsm will fire events using json-rpc
notifications sent to the clients. The will be in the form of:
{"method": "alert", "params": {
"alertName": (.)?..(., ...),
"operationInfo", OperationInfo}
}
The user can register to recive events using a glob or a regexp.
registering to vdsm.VM.* pop every time any VM has changed stage.
This means that whenever the task finishes, fails or gains significance progress
and VDSM is there to track it, an event will be sent to the client.
This means that the general flow is.
# Register operation
vmID = best_vm
host.VM.pauseVM(vmID)
while True:
opInfo = None
try:
event = host.waitForEvent("vdsm.VM.best_vm", timeout=10)
opInfo = event.opInfo
except VdsmDisconnectionError:
host.waitForReconnect()
host.vm.getVmInfo(vmID) # Double check that we didn't miss the event
continue
except Timeout:
# This is a long operation, poll to see that we didn't miss any event
# but more commonly, update percentage in the UI to show progress.
vmInfo = host.vm.getVmInfo(vmID)
opInfo = vmInfo.operationInfo
if opInfo.stage.number != op.stage.total:
# Operation in progress
updateUI(opInfo)
else:
# Operation completed
# Check that the state is what we expected it to be.
if oInfo.extraData.state == "paused":
return SUCCESS
else:
Re: [vdsm] Managing async tasks
- Original Message - > This is an addendum to my previous email. > > - Original Message - > > From: "Saggi Mizrahi" > > To: "Adam Litke" > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > , "Federico Simoncelli" > > , engine-de...@ovirt.org, > > vdsm-devel@lists.fedorahosted.org > > Sent: Monday, December 17, 2012 2:52:06 PM > > Subject: Re: Managing async tasks > > > > > > > > - Original Message - > > > From: "Adam Litke" > > > To: "Saggi Mizrahi" > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > , "Federico Simoncelli" > > > , engine-de...@ovirt.org, > > > vdsm-devel@lists.fedorahosted.org > > > Sent: Monday, December 17, 2012 2:16:25 PM > > > Subject: Re: Managing async tasks > > > > > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > > > > > > > > > > - Original Message - > > > > > From: "Adam Litke" To: > > > > > vdsm-devel@lists.fedorahosted.org > > > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > > > , > > > > > "Saggi Mizrahi" , "Federico Simoncelli" > > > > > , engine-de...@ovirt.org Sent: Monday, > > > > > December 17, > > > > > 2012 12:00:49 PM Subject: Managing async tasks > > > > > > > > > > On today's vdsm call we had a lively discussion around how > > > > > asynchronous > > > > > operations should be handled in the future. In an effort to > > > > > include more > > > > > people in the discussion and to better capture the resulting > > > > > conversation I > > > > > would like to continue that discussion here on the mailing > > > > > list. > > > > > > > > > > A lot of ideas were thrown around about how 'tasks' should be > > > > > handled in the > > > > > future. There are a lot of ways that it can be done. To > > > > > determine how we > > > > > should implement it, it's probably best if we start with a > > > > > set > > > > > of > > > > > requirements. If we can first agree on these, it should be > > > > > easy > > > > > to find a > > > > > solution that meets them. I'll take a stab at identifying a > > > > > first set of > > > > > POSSIBLE requirements: > > > > > > > > > > - Standardized method for determining the result of an > > > > > operation > > > > > > > > > > This is a big one for me because it directly affects the > > > > > consumability of > > > > > the API. If each verb has different semantics for > > > > > discovering > > > > > whether it > > > > > has completed successfully, then the API will be nearly > > > > > impossible to use > > > > > easily. > > > > Since there is no way to assure if of some tasks completed > > > > successfully or > > > > failed, especially around the murky waters of storage, I say > > > > this > > > > requirement > > > > should be removed. At least not in the context of a task. > > > > > > I don't agree. Please feel free to convince me with some > > > exampled. > > > If we > > > cannot provide feedback to a user as to whether their request has > > > been satisfied > > > or not, then we have some bigger problems to solve. > > If VDSM sends a write command to a storage server, and the > > connection > > hangs up before the ACK has returned. > > The operation has been committed but VDSM has no way of knowing if > > that happened as far as VDSM is concerned it got an ETIMEO or EIO. > > This is the same problem that the engine has with VDSM. > > If VDSM creates an image\VM\network\repo but the connection hangs > > up > > before the response can be sent back as far as the engine is > > concerned the operation times out. > > This is an inherent issue with clustering. > > This is why I want to move away from tasks being *the* trackable > > objects. > > Tasks should be short. As short as possible. > > Run VM should just persist the VM information on the VDSM host and > > return. The rest of the tracking should be done using the VM ID. > > Create image should return once VDSM persisted the information > > about > > the request on the repository and created the metadata files. > > Tracking should be done on the repo or the imageId. > > The thing is that I know how long a VM object should live (or an > Image object). > So tracking it is straight forward. How long a task should live is > very problematic and quite context specific. > It depends on what the task is. > I think it's quite confusing from an API standpoint to have every > task have a different scope, id requirement and life-cycle. > > In VDSM has two types of APIs > > CRUD objects - VM, Image, Repository, Bridge, Storage Connections > General transient methods - getBiosInfo(), getDeviceList() > > The latter are quite simple to manage. They don't need any special > handling. If you lost a getBiosInfo() call you just send another > one, no harm done. > The same is even true with things that "change" the host like > getDeviceList() > > What we are really arguing about is fitting the CRUD objects to some > generic task oriented scheme. > I'm saying it's a waste of time as you can quite easily have flows to > recover
Re: [vdsm] Managing async tasks
This is an addendum to my previous email. - Original Message - > From: "Saggi Mizrahi" > To: "Adam Litke" > Cc: "Dan Kenigsberg" , "Ayal Baron" , > "Federico Simoncelli" > , engine-de...@ovirt.org, > vdsm-devel@lists.fedorahosted.org > Sent: Monday, December 17, 2012 2:52:06 PM > Subject: Re: Managing async tasks > > > > - Original Message - > > From: "Adam Litke" > > To: "Saggi Mizrahi" > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > , "Federico Simoncelli" > > , engine-de...@ovirt.org, > > vdsm-devel@lists.fedorahosted.org > > Sent: Monday, December 17, 2012 2:16:25 PM > > Subject: Re: Managing async tasks > > > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > > > > > > > - Original Message - > > > > From: "Adam Litke" To: > > > > vdsm-devel@lists.fedorahosted.org > > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > > , > > > > "Saggi Mizrahi" , "Federico Simoncelli" > > > > , engine-de...@ovirt.org Sent: Monday, > > > > December 17, > > > > 2012 12:00:49 PM Subject: Managing async tasks > > > > > > > > On today's vdsm call we had a lively discussion around how > > > > asynchronous > > > > operations should be handled in the future. In an effort to > > > > include more > > > > people in the discussion and to better capture the resulting > > > > conversation I > > > > would like to continue that discussion here on the mailing > > > > list. > > > > > > > > A lot of ideas were thrown around about how 'tasks' should be > > > > handled in the > > > > future. There are a lot of ways that it can be done. To > > > > determine how we > > > > should implement it, it's probably best if we start with a set > > > > of > > > > requirements. If we can first agree on these, it should be > > > > easy > > > > to find a > > > > solution that meets them. I'll take a stab at identifying a > > > > first set of > > > > POSSIBLE requirements: > > > > > > > > - Standardized method for determining the result of an > > > > operation > > > > > > > > This is a big one for me because it directly affects the > > > > consumability of > > > > the API. If each verb has different semantics for > > > > discovering > > > > whether it > > > > has completed successfully, then the API will be nearly > > > > impossible to use > > > > easily. > > > Since there is no way to assure if of some tasks completed > > > successfully or > > > failed, especially around the murky waters of storage, I say this > > > requirement > > > should be removed. At least not in the context of a task. > > > > I don't agree. Please feel free to convince me with some exampled. > > If we > > cannot provide feedback to a user as to whether their request has > > been satisfied > > or not, then we have some bigger problems to solve. > If VDSM sends a write command to a storage server, and the connection > hangs up before the ACK has returned. > The operation has been committed but VDSM has no way of knowing if > that happened as far as VDSM is concerned it got an ETIMEO or EIO. > This is the same problem that the engine has with VDSM. > If VDSM creates an image\VM\network\repo but the connection hangs up > before the response can be sent back as far as the engine is > concerned the operation times out. > This is an inherent issue with clustering. > This is why I want to move away from tasks being *the* trackable > objects. > Tasks should be short. As short as possible. > Run VM should just persist the VM information on the VDSM host and > return. The rest of the tracking should be done using the VM ID. > Create image should return once VDSM persisted the information about > the request on the repository and created the metadata files. > Tracking should be done on the repo or the imageId. The thing is that I know how long a VM object should live (or an Image object). So tracking it is straight forward. How long a task should live is very problematic and quite context specific. It depends on what the task is. I think it's quite confusing from an API standpoint to have every task have a different scope, id requirement and life-cycle. In VDSM has two types of APIs CRUD objects - VM, Image, Repository, Bridge, Storage Connections General transient methods - getBiosInfo(), getDeviceList() The latter are quite simple to manage. They don't need any special handling. If you lost a getBiosInfo() call you just send another one, no harm done. The same is even true with things that "change" the host like getDeviceList() What we are really arguing about is fitting the CRUD objects to some generic task oriented scheme. I'm saying it's a waste of time as you can quite easily have flows to recover from each operation. Create - Check if the object exists Read - Read again Update - either update again or read and update if update didn't commit the first time Delete - Check if object doesn't exist Each of the objects we CRUD have different life-cycles and ownership semantics. Danken raised the
Re: [vdsm] Managing async tasks
- Original Message - > From: "Adam Litke" > To: "Saggi Mizrahi" > Cc: "Dan Kenigsberg" , "Ayal Baron" , > "Federico Simoncelli" > , engine-de...@ovirt.org, > vdsm-devel@lists.fedorahosted.org > Sent: Monday, December 17, 2012 2:16:25 PM > Subject: Re: Managing async tasks > > On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > > > > - Original Message - > > > From: "Adam Litke" To: > > > vdsm-devel@lists.fedorahosted.org > > > Cc: "Dan Kenigsberg" , "Ayal Baron" > > > , > > > "Saggi Mizrahi" , "Federico Simoncelli" > > > , engine-de...@ovirt.org Sent: Monday, > > > December 17, > > > 2012 12:00:49 PM Subject: Managing async tasks > > > > > > On today's vdsm call we had a lively discussion around how > > > asynchronous > > > operations should be handled in the future. In an effort to > > > include more > > > people in the discussion and to better capture the resulting > > > conversation I > > > would like to continue that discussion here on the mailing list. > > > > > > A lot of ideas were thrown around about how 'tasks' should be > > > handled in the > > > future. There are a lot of ways that it can be done. To > > > determine how we > > > should implement it, it's probably best if we start with a set of > > > requirements. If we can first agree on these, it should be easy > > > to find a > > > solution that meets them. I'll take a stab at identifying a > > > first set of > > > POSSIBLE requirements: > > > > > > - Standardized method for determining the result of an operation > > > > > > This is a big one for me because it directly affects the > > > consumability of > > > the API. If each verb has different semantics for discovering > > > whether it > > > has completed successfully, then the API will be nearly > > > impossible to use > > > easily. > > Since there is no way to assure if of some tasks completed > > successfully or > > failed, especially around the murky waters of storage, I say this > > requirement > > should be removed. At least not in the context of a task. > > I don't agree. Please feel free to convince me with some exampled. > If we > cannot provide feedback to a user as to whether their request has > been satisfied > or not, then we have some bigger problems to solve. If VDSM sends a write command to a storage server, and the connection hangs up before the ACK has returned. The operation has been committed but VDSM has no way of knowing if that happened as far as VDSM is concerned it got an ETIMEO or EIO. This is the same problem that the engine has with VDSM. If VDSM creates an image\VM\network\repo but the connection hangs up before the response can be sent back as far as the engine is concerned the operation times out. This is an inherent issue with clustering. This is why I want to move away from tasks being *the* trackable objects. Tasks should be short. As short as possible. Run VM should just persist the VM information on the VDSM host and return. The rest of the tracking should be done using the VM ID. Create image should return once VDSM persisted the information about the request on the repository and created the metadata files. Tracking should be done on the repo or the imageId. > > > > > > > > > > Sorry. That's my list :) Hopefully others will be willing to > > > add other > > > requirements for consideration. > > > > > > From my understanding, task recovery (stop, abort, rollback, etc) > > > will not > > > be generally supported and should not be a requirement. > > > > > -- > Adam Litke > IBM Linux Technology Center > > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Managing async tasks
On Mon, Dec 17, 2012 at 12:15:08PM -0500, Saggi Mizrahi wrote: > > > - Original Message - > > From: "Adam Litke" To: vdsm-devel@lists.fedorahosted.org > > Cc: "Dan Kenigsberg" , "Ayal Baron" , > > "Saggi Mizrahi" , "Federico Simoncelli" > > , engine-de...@ovirt.org Sent: Monday, December 17, > > 2012 12:00:49 PM Subject: Managing async tasks > > > > On today's vdsm call we had a lively discussion around how asynchronous > > operations should be handled in the future. In an effort to include more > > people in the discussion and to better capture the resulting conversation I > > would like to continue that discussion here on the mailing list. > > > > A lot of ideas were thrown around about how 'tasks' should be handled in the > > future. There are a lot of ways that it can be done. To determine how we > > should implement it, it's probably best if we start with a set of > > requirements. If we can first agree on these, it should be easy to find a > > solution that meets them. I'll take a stab at identifying a first set of > > POSSIBLE requirements: > > > > - Standardized method for determining the result of an operation > > > > This is a big one for me because it directly affects the consumability of > > the API. If each verb has different semantics for discovering whether it > > has completed successfully, then the API will be nearly impossible to use > > easily. > Since there is no way to assure if of some tasks completed successfully or > failed, especially around the murky waters of storage, I say this requirement > should be removed. At least not in the context of a task. I don't agree. Please feel free to convince me with some exampled. If we cannot provide feedback to a user as to whether their request has been satisfied or not, then we have some bigger problems to solve. > > > > > > Sorry. That's my list :) Hopefully others will be willing to add other > > requirements for consideration. > > > > From my understanding, task recovery (stop, abort, rollback, etc) will not > > be generally supported and should not be a requirement. > > -- Adam Litke IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Managing async tasks
- Original Message - > From: "Adam Litke" > To: vdsm-devel@lists.fedorahosted.org > Cc: "Dan Kenigsberg" , "Ayal Baron" , > "Saggi Mizrahi" , > "Federico Simoncelli" , engine-de...@ovirt.org > Sent: Monday, December 17, 2012 12:00:49 PM > Subject: Managing async tasks > > On today's vdsm call we had a lively discussion around how > asynchronous > operations should be handled in the future. In an effort to include > more people > in the discussion and to better capture the resulting conversation I > would like > to continue that discussion here on the mailing list. > > A lot of ideas were thrown around about how 'tasks' should be handled > in the > future. There are a lot of ways that it can be done. To determine > how we > should implement it, it's probably best if we start with a set of > requirements. > If we can first agree on these, it should be easy to find a solution > that meets > them. I'll take a stab at identifying a first set of POSSIBLE > requirements: > > - Standardized method for determining the result of an operation > > This is a big one for me because it directly affects the > consumability of the > API. If each verb has different semantics for discovering whether > it has > completed successfully, then the API will be nearly impossible to > use easily. Since there is no way to assure if of some tasks completed successfully or failed, especially around the murky waters of storage, I say this requirement should be removed. At least not in the context of a task. > > > Sorry. That's my list :) Hopefully others will be willing to add > other > requirements for consideration. > > From my understanding, task recovery (stop, abort, rollback, etc) > will not be > generally supported and should not be a requirement. > > > > -- > Adam Litke > IBM Linux Technology Center > > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] Managing async tasks
On today's vdsm call we had a lively discussion around how asynchronous operations should be handled in the future. In an effort to include more people in the discussion and to better capture the resulting conversation I would like to continue that discussion here on the mailing list. A lot of ideas were thrown around about how 'tasks' should be handled in the future. There are a lot of ways that it can be done. To determine how we should implement it, it's probably best if we start with a set of requirements. If we can first agree on these, it should be easy to find a solution that meets them. I'll take a stab at identifying a first set of POSSIBLE requirements: - Standardized method for determining the result of an operation This is a big one for me because it directly affects the consumability of the API. If each verb has different semantics for discovering whether it has completed successfully, then the API will be nearly impossible to use easily. Sorry. That's my list :) Hopefully others will be willing to add other requirements for consideration. From my understanding, task recovery (stop, abort, rollback, etc) will not be generally supported and should not be a requirement. -- Adam Litke IBM Linux Technology Center ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Meeting minutes
> - When adding a VM network on top of a bond device, we take the > relevant > devices down, we write their new ifcfg-* files (with > NM_CONTROLLED=no), and take them up again. > At this point, NM notices that the devices are no longer under its > management, and takes them down asynchronously. See > https://bugzilla.redhat.com/879180 . > This race may force us to turn NM off on installation, but we'd > rather > have a finer-grain work-around. Pavel, maybe you can help us here. Sure. For the nearest future... You should probably first synchronously unmanage it. I'm not sure whether there's API for that but it should would not be that hard to add that. As a long-term solution, you should *never* write ifcfg files. You should use network management APIs for your configurations instead. The ifcfg files are distribution specific and may not be supported in some current or future networkmanager installations. Adding Jirka to Cc as he knows more about the available D-Bus interfaces. Cheers, Pavel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] Meeting minutes
Speaking attendees (no particular order): Dustin, Saggi, Adam, Ayal, Toni, Federico, Danken. Issues raised: - Danken thanks Adam for pulling active developers from Beijing to #v...@irc.freenode.net . It is fun to chat on irc, it's quick, and may be fruitful. So please autoconnect to #vdsm when you turn your desktop on! - ovirt-3.2 release: nothing urgent, appart of a NetworkManager integration issue (see bellow). If there is a show stopper, please rebase to the ovirt-3.2 branch, and ping Federico. - When adding a VM network on top of a bond device, we take the relevant devices down, we write their new ifcfg-* files (with NM_CONTROLLED=no), and take them up again. At this point, NM notices that the devices are no longer under its management, and takes them down asynchronously. See https://bugzilla.redhat.com/879180 . This race may force us to turn NM off on installation, but we'd rather have a finer-grain work-around. Pavel, maybe you can help us here. - Task framework: we've spend a lot of time discussing with Saggi what a task framework should not be. According to Saggi, Engine can never expect to know that a task has finished, since it may loose connection to the host running it. Thus, in the worst case, Engine has to be able to poll for whatever entity was created/destroyed by that task. Saggi suggests to make this worst case the only case. He prefer to keep the current situation, where we have different verbs to create and poll for vm creation, start and poll migration progression, etc, and extend it to gluster tasks and storage image creation/deletion/copying tasks. To me it seems that an end user would benefit of having a unified view of what is currently going on in vdsm, how it progresses, and whether it can be stopped. However, this can be wrapped up nicely within the client with no abstraction in Vdsm. I may have well lost part of the discussion, so please correct me on list I I misrepresented an opinion. Regards, Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] RFC: New Storage API
- Original Message -
> From: "Deepak C Shetty"
> To: "Saggi Mizrahi"
> Cc: "Shu Ming" , "engine-devel"
> , "VDSM Project Development"
> , "Deepak C Shetty"
>
> Sent: Sunday, December 16, 2012 11:40:01 PM
> Subject: Re: [vdsm] RFC: New Storage API
>
> On 12/08/2012 01:23 AM, Saggi Mizrahi wrote:
> >
> > - Original Message -
> >> From: "Deepak C Shetty"
> >> To: "Saggi Mizrahi"
> >> Cc: "Shu Ming" , "engine-devel"
> >> , "VDSM Project Development"
> >> , "Deepak C Shetty"
> >>
> >> Sent: Friday, December 7, 2012 12:23:15 AM
> >> Subject: Re: [vdsm] RFC: New Storage API
> >>
> >> On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:
> >>> - Original Message -
> From: "Shu Ming"
> To: "Saggi Mizrahi"
> Cc: "VDSM Project Development"
> , "engine-devel"
>
> Sent: Thursday, December 6, 2012 11:02:02 AM
> Subject: Re: [vdsm] RFC: New Storage API
>
> Saggi,
>
> Thanks for sharing your thought and I get some comments below.
>
>
> Saggi Mizrahi:
> > I've been throwing a lot of bits out about the new storage API
> > and
> > I think it's time to talk a bit.
> > I will purposefully try and keep implementation details away
> > and
> > concentrate about how the API looks and how you use it.
> >
> > First major change is in terminology, there is no long a
> > storage
> > domain but a storage repository.
> > This change is done because so many things are already called
> > domain in the system and this will make things less confusing
> > for
> > new-commers with a libvirt background.
> >
> > One other changes is that repositories no longer have a UUID.
> > The UUID was only used in the pool members manifest and is no
> > longer needed.
> >
> >
> > connectStorageRepository(repoId, repoFormat,
> > connectionParameters={}):
> > repoId - is a transient name that will be used to refer to the
> > connected domain, it is not persisted and doesn't have to be
> > the
> > same across the cluster.
> > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > nfs-3.4, clvm-1.2).
> > connectionParameters - This is format specific and will used to
> > tell VDSM how to connect to the repo.
> Where does repoID come from? I think repoID doesn't exist before
> connectStorageRepository() return. Isn't repoID a return value
> of
> connectStorageRepository()?
> >>> No, repoIDs are no longer part of the domain, they are just a
> >>> transient handle.
> >>> The user can put whatever it wants there as long as it isn't
> >>> already taken by another currently connected domain.
> >> So what happens when user mistakenly gives a repoID that is in use
> >> before.. there should be something in the return value that
> >> specifies
> >> the error and/or reason for error so that user can try with a
> >> new/diff
> >> repoID ?
> > Asi I said, connect fails if the repoId is in use ATM.
> > disconnectStorageRepository(self, repoId)
> >
> >
> > In the new API there are only images, some images are mutable
> > and
> > some are not.
> > mutable images are also called VirtualDisks
> > immutable images are also called Snapshots
> >
> > There are no explicit templates, you can create as many images
> > as
> > you want from any snapshot.
> >
> > There are 4 major image operations:
> >
> >
> > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> > userData={}, options={}):
> >
> > targetRepoId - ID of a connected repo where the disk will be
> > created
> > size - The size of the image you wish to create
> > baseSnapshotId - the ID of the snapshot you want the base the
> > new
> > virtual disk on
> > userData - optional data that will be attached to the new VD,
> > could
> > be anything that the user desires.
> > options - options to modify VDSMs default behavior
> >> IIUC, i can use options to do storage offloads ? For eg. I can
> >> create
> >> a
> >> LUN that represents this VD on my storage array based on the
> >> 'options'
> >> parameter ? Is this the intended way to use 'options' ?
> > No, this has nothing to do with offloads.
> > If by "offloads" you mean having other VDSM hosts to the heavy
> > lifting then this is what the option autoFix=False and the fix
> > mechanism is for.
> > If you are talking about advanced scsi features (ie. write same)
> > they will be used automatically whenever possible.
> > In any case, how we manage LUNs (if they are even used) is an
> > implementation detail.
>
> I am a bit more interested in how storage array offloads ( by that I
> mean, offload VD creation, snapshot, clone etc to the storage array
> when
> available/possible) can be done from VDSM ?
> In the past there were talks of using libSM to do that. How does that
> strategy play in
