Re: [vdsm] Verify the storage data integrity after some storage operations with test cases
Well libguestfs really uses a VM behind the scenes, so it's the same thing only less work For online snapshotting, I agree this is a problem, but online snapshotting is only 1 test. I think most tests that can should use libguestfs, it means that they theoretically could be done without a full blown VDSM running. - Original Message - > From: "Shu Ming" > To: "Saggi Mizrahi" > Cc: "VDSM Project Development" > Sent: Tuesday, July 17, 2012 11:48:39 AM > Subject: Re: [vdsm] Verify the storage data integrity after some storage > operations with test cases > > On 2012-7-17 23:21, Saggi Mizrahi wrote: > > Actually setting up isos and installing an OS is an overkill IMHO. > > Using libguestfs seems simpler as it has python bindings. > > > > What you could do is: > > 1. use libguest fs to format a file system on an image > Do you mean the test case use libguest fs to format a file without > interaction with VM? > > > 2. Put files on said file system with libguestfs > Do you mean the test case put files to the files system with > libguestfs > directly without interaction with VM? > Or even VM is not running? > > > 3. Snapshot > If we want to test online snapshot, we should have a running VM now. > > > 4. run fsck with libguestfs > > 5. rinse > > 6. repeast > > > > If you don't trust fsck to detect all issues you can use libguestfs > > to get an md5sum of the raw drive and make sure that after a > > snapshot it stays the same. > > > > > > - Original Message - > >> From: "Shu Ming" > >> To: "VDSM Project Development" > >> Sent: Monday, July 16, 2012 10:28:25 PM > >> Subject: [vdsm] Verify the storage data integrity after some > >> storage operations with test cases > >> > >> Hi, > >> > >>To verify the storage data integrity after some storage > >>operations > >> like snapshot, merging by VDSM. Here are the test cases I am > >> pondering. > >> I would like to know your feedback about these thoughts. > >> > >> 1) An customized ISO image with the agent required prepared for > >> bringing up a VM in VDSM > >> 2) The test case will inform VDSM to create a VM from the > >> customized > >> ISO > >> image > >> 3) The test case will install an IO application to the VM > >> 3) The test case communicate with the VDSM to inform the IO > >> application > >> in the VM to write some data intentionally. > >> 4) The test case sends the commands to VDSM do some storage > >> operation > >> like disk snapshot, volume merging, &etc. > >>Say snapshot operation here for an example. > >> 5) VDSM then tell the test case the result of the operation like > >> the > >> name of the snapshot. > >> 6) Test case can read the snapshot made to verify the snapshot > >> with > >> the > >> data written in 3). > >>Note: currently, there is no tool to read the snapshot > >>image > >> directly. We can restart the VM with the snapshot as > >>the active disk and tell the IO application in the VM to > >>read > >>the > >> data writen before for test case. And test case can compare > >> the data read with the data it informs the application in > >> 3). > >> 7) If the two data matches, the storage operation succeed or it > >> fails. > >> > >> In order to write such a test case, these VDSM features will be > >> required. > >> > >> 1) VDSM can create a VM from a specific ISO image (Almost works) > >> 2) Test case can install an IO application to the VM by VDSM (by > >> ovirt-agent?) > >> 3) Test case must have some protocols with the IO application in > >> VM > >> for > >> passing the command to the VM and returning the result from the VM > >>to the test case(by ovirt-agent?). > >> 4) The IO application can be seen as an test agent. We may extend > >> the > >> existing agent like ovirt-agent as the IO application. > >> > >> > >> -- > >> Shu Ming > >> IBM China Systems and Technology Laboratory > >> > >> > >> ___ > >> vdsm-devel mailing list > >> vdsm-devel@lists.fedorahosted.org > >> https://fedorahosted.org/mailman/listinfo/vdsm-devel > >> > > > -- > Shu Ming > IBM China Systems and Technology Laboratory > > > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Verify the storage data integrity after some storage operations with test cases
On 2012-7-17 23:21, Saggi Mizrahi wrote: Actually setting up isos and installing an OS is an overkill IMHO. Using libguestfs seems simpler as it has python bindings. What you could do is: 1. use libguest fs to format a file system on an image Do you mean the test case use libguest fs to format a file without interaction with VM? 2. Put files on said file system with libguestfs Do you mean the test case put files to the files system with libguestfs directly without interaction with VM? Or even VM is not running? 3. Snapshot If we want to test online snapshot, we should have a running VM now. 4. run fsck with libguestfs 5. rinse 6. repeast If you don't trust fsck to detect all issues you can use libguestfs to get an md5sum of the raw drive and make sure that after a snapshot it stays the same. - Original Message - From: "Shu Ming" To: "VDSM Project Development" Sent: Monday, July 16, 2012 10:28:25 PM Subject: [vdsm] Verify the storage data integrity after some storage operations with test cases Hi, To verify the storage data integrity after some storage operations like snapshot, merging by VDSM. Here are the test cases I am pondering. I would like to know your feedback about these thoughts. 1) An customized ISO image with the agent required prepared for bringing up a VM in VDSM 2) The test case will inform VDSM to create a VM from the customized ISO image 3) The test case will install an IO application to the VM 3) The test case communicate with the VDSM to inform the IO application in the VM to write some data intentionally. 4) The test case sends the commands to VDSM do some storage operation like disk snapshot, volume merging, &etc. Say snapshot operation here for an example. 5) VDSM then tell the test case the result of the operation like the name of the snapshot. 6) Test case can read the snapshot made to verify the snapshot with the data written in 3). Note: currently, there is no tool to read the snapshot image directly. We can restart the VM with the snapshot as the active disk and tell the IO application in the VM to read the data writen before for test case. And test case can compare the data read with the data it informs the application in 3). 7) If the two data matches, the storage operation succeed or it fails. In order to write such a test case, these VDSM features will be required. 1) VDSM can create a VM from a specific ISO image (Almost works) 2) Test case can install an IO application to the VM by VDSM (by ovirt-agent?) 3) Test case must have some protocols with the IO application in VM for passing the command to the VM and returning the result from the VM to the test case(by ovirt-agent?). 4) The IO application can be seen as an test agent. We may extend the existing agent like ovirt-agent as the IO application. -- Shu Ming IBM China Systems and Technology Laboratory ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel -- Shu Ming IBM China Systems and Technology Laboratory ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] How should we handle aborted tasks? via engine, vdsClient or both?
On 07/13/2012 02:53 PM, Lee Yarwood wrote:
> Hello all,
>
> I'm looking into the use case around an aborted task within vdsmd.
> AFAICT both engine and vdsClient are unable to deal with an aborted task
> at present.
>
> Within engine we call HSMStopTaskVDSCommand, this inturn calls stopTask
> supplying _only_ the task_guid. vdsClient also calls stopTask with only
> the task_guid argument.
>
> Why does this matter? Well the only was you can kill an aborted task is
> by passing the force=True argument to the stop method of the task object
> itself. Without this the _incref method will throw a TaskAborted
> exception and we will be unable to stop the task.
>
> Could anyone give me some background on why this is the case before I
> start looking into exposing the force option up the stack to engine and
> vdsClient.
>
> Relevant code snippets from task.py below.
>
> Thanks in advance,
>
> Lee
>
> vdsm/storage/task.py
>
> 1198 def stop(self, force=False):
> 1199 self.log.debug("stopping in state %s (force %s)",
> self.state, force)
> 1200 self._incref(force)
> 1201 try:
> 1202 if self.state.isDone():
> 1203 self.log.debug("Task already stopped (%s),
> ignoring", self.state)
> 1204 return
> 1205 elif self.state.isRecovering() and not force and
> (self.cleanPolicy == TaskCleanType.auto):
> 1206 self.log.debug("Task (%s) in recovery and force is
> false, ignoring", self.state)
> 1207 return
> 1208
> 1209 self._aborting = True
> 1210 self._forceAbort = force
> 1211 finally:
> 1212 self._decref(force)
>
> 959 def _incref(self, force=False):
> 960 self.lock.acquire()
> 961 try:
> 962 if self.aborting() and (self._forceAbort or not force):
> 963 raise se.TaskAborted(unicode(self))
> 964
> 965 self.ref += 1
> 966 ref = self.ref
> 967 return ref
> 968 finally:
> 969 self.lock.release()
>
Bump, any thoughts? Apologies if this is a little ovirt-engine centric
for the list.
Lee
--
Lee Yarwood
Software Maintenance Engineer
Red Hat UK Ltd
200 Fowler Avenue IQ Farnborough, Farnborough, Hants GU14 7JP
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham (USA), Brendan Lane (Ireland), Matt
Parson(USA), Charlie Peters (USA)
GPG fingerprint : A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Verify the storage data integrity after some storage operations with test cases
Actually setting up isos and installing an OS is an overkill IMHO. Using libguestfs seems simpler as it has python bindings. What you could do is: 1. use libguest fs to format a file system on an image 2. Put files on said file system with libguestfs 3. Snapshot 4. run fsck with libguestfs 5. rinse 6. repeast If you don't trust fsck to detect all issues you can use libguestfs to get an md5sum of the raw drive and make sure that after a snapshot it stays the same. - Original Message - > From: "Shu Ming" > To: "VDSM Project Development" > Sent: Monday, July 16, 2012 10:28:25 PM > Subject: [vdsm] Verify the storage data integrity after some storage > operations with test cases > > Hi, > > To verify the storage data integrity after some storage operations > like snapshot, merging by VDSM. Here are the test cases I am > pondering. > I would like to know your feedback about these thoughts. > > 1) An customized ISO image with the agent required prepared for > bringing up a VM in VDSM > 2) The test case will inform VDSM to create a VM from the customized > ISO > image > 3) The test case will install an IO application to the VM > 3) The test case communicate with the VDSM to inform the IO > application > in the VM to write some data intentionally. > 4) The test case sends the commands to VDSM do some storage operation > like disk snapshot, volume merging, &etc. > Say snapshot operation here for an example. > 5) VDSM then tell the test case the result of the operation like the > name of the snapshot. > 6) Test case can read the snapshot made to verify the snapshot with > the > data written in 3). > Note: currently, there is no tool to read the snapshot image > directly. We can restart the VM with the snapshot as > the active disk and tell the IO application in the VM to read > the > data writen before for test case. And test case can compare >the data read with the data it informs the application in 3). > 7) If the two data matches, the storage operation succeed or it > fails. > > In order to write such a test case, these VDSM features will be > required. > > 1) VDSM can create a VM from a specific ISO image (Almost works) > 2) Test case can install an IO application to the VM by VDSM (by > ovirt-agent?) > 3) Test case must have some protocols with the IO application in VM > for > passing the command to the VM and returning the result from the VM > to the test case(by ovirt-agent?). > 4) The IO application can be seen as an test agent. We may extend > the > existing agent like ovirt-agent as the IO application. > > > -- > Shu Ming > IBM China Systems and Technology Laboratory > > > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/vdsm-devel > ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] A few notes about lists in Makefiles
On Mon, Jul 16, 2012 at 11:34:34AM -0400, Saggi Mizrahi wrote: > Hi, I would just like to push a few notes to people modifying > autoconf\automake lists > > Please make sure the lists are sorted. Sorted lists are easier to skim and > modify. > Also, unsorted lists are known to make Federico sad, and we all want to keep > him happy because he is a pretty swell guy and the one that we actually have > to thank for the amazing build system. > > Also please make sure to add the $(NULL) item so when auto sorting you don't > need to check if you need to add\remove a backslash. > > VARIABLE = \ >A \ >B \ >C \ >$(NULL) Ooh, $(NULL) is a cool trick. I've had my share of broken ovirt/master due to a bad rebase leading to a trailing backslash. > > If you are using vim you could just mark all the lines an run ":<>!sort" to > sort. > > Also, when adding a file to the PEP_WHITELIST, check if you can just mark the > entire directory instead of the individual file. > > Remember, cleanliness is next to godliness. > ___ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
