Hi Jeremy,
If you revisit the info on vhd-util this will still help you troubleshoot LVM
based storage. Another KB article you can refer to is
https://support.citrix.com/article/CTX201296 - as you can see from this the
scans will work for both (e.g. SIC #vhd-util scan –m “VHD-*” -f -c -l
VG_XenStorage-<SR-UUID> -p –v}
In short though – if your base disk is gone I’m not aware of any mechanism to
recover data from your delta disk.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 25/09/2017, 18:02, "Jeremy Peterson" <jpeter...@acentek.net> wrote:
Now looking at Adrian's link
Since the VM is powered off and will not boot due to UUID mismatch.
[root@Flex-Xen5 mapper]# xe vm-list name-label=i-23-223-VM
I don't get anything after issuing the vm-list command for that VM.
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Monday, September 25, 2017 11:55 AM
To: users@cloudstack.apache.org
Subject: RE: Corrupted HDD
[root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe
vdi-list | grep -A 5 12eb0cdaba30
[root@Flex-Xen5 VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# xe
vdi-list | grep -A 5 d750a907a699
You can see above that both uuid's I pulled one from the uuid not found
command from cloudstack and also from the sql query both are not showing
anything when I do a xe vdi-list. This tells me that xenserver cannot find the
vdi's leaving me with only my block device.
[root@Flex-Xen5 mapper]# ls -lh | grep 699
brw-rw---- 1 root disk 253, 9 Sep 23 11:34
VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
Sorry I am responding so many times I'm just going through this one by one
trying to figure anything out to recover this data.
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Monday, September 25, 2017 11:44 AM
To: users@cloudstack.apache.org
Subject: RE: Corrupted HDD
Now if I look at my starting error from cloudstack I see the following for
when it calls the ROOT disk.
"disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
"uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30",
"volumeType":"ROOT",
2017-09-25 11:20:16,555 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-57:ctx-dcbad35d job-261776/job-261778 ctx-f1843e45) Seq
19-8416664753602063711: Executing: { Cmd , MgmtId:
345050411715, via: 19(Flex-Xen1.flexhost.local), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StartCommand":{"vm":{"id":223,"name":"i-23-223-VM","bootloader":"PyGrub","typ
e":"User","cpus":2,"minSpeed":500,"maxSpeed":2000,"minRam":1073741824,"maxRam":4294967296,"arch":"x86_64","os":"Windows
Server 2012 R2 (64-bit)","platformEmulator":"Windows S
erver 2012 R2
(64-bit)","bootArgs":"","enableHA":true,"limitCpuUse":true,"enableDynamicallyScaleVm":true,"vncPassword":"P2LDsJVeaBNz2+W4OB93ZA==","params":{"memoryOvercommitR
atio":"4","platform":"viridian:true;acpi:1;apic:true;viridian_reference_tsc:true;viridian_time_ref_count:true;pae:true;videoram:8;device_id:0002;nx:true;timeoffset:-17885;vga
:std","keyboard":"us","Message.ReservedCapacityFreed.Flag":"false","cpuOvercommitRatio":"4","hypervisortoolsversion":"xenserver61"},"uuid":"6ff4beb0-e064-4364-94d3-5b3fbc15ee
16","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e772d773-5a9d-4c7b-9227-12eb0cdaba30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack
.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN1","id":6,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN1","port":0,"url":"PreSetup://localhost/FlexSAN2-
LUN1/?ROLE=Primary&STOREUUID=FlexSAN2-LUN1"}},"name":"ROOT-223","size":171798691840,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","volumeId":275,"vmName":"i-23-223-VM","accou
ntId":23,"format":"VHD","provisioningType":"THIN","id":275,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"f5aabea5-2a18-4f24-9f8d-d750a907a699","type":"ROOT"
,"_details":{"managed":"false","storagePort":"0","storageHost":"localhost","volumeSize":"171798691840"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,
"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"defaultNic":true,"pxeDisable":false,"nicUuid":"d7d464bc-64
c1-490a-944e-6a2adee1a451","uuid":"940feda7-65ee-438b-bd06-d33ba3d734c4","ip":"10.1.1.126","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:62:78:00:06","dns1":"20
8.74.240.5","dns2":"208.74.247.245","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1679","isolationUri":"vlan://1679","isSecurityGroupEnabled":false,"name":"GUE
ST-PUB"}],"vcpuMaxLimit":16},"hostIp":"10.90.2.111","executeInSequence":false,"wait":0}}]
}
2017-09-25 11:20:16,555 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-378:ctx-d30ac189) Seq 19-8416664753602063711: Executing request
2017-09-25 11:20:16,560 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-378:ctx-d30ac189) 1. The VM i-23-223-VM is in Starting state.
2017-09-25 11:20:16,579 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-378:ctx-d30ac189) Created VM a6739bcc-968f-abe6-d85c-a016de8b0b9f
for i-23-223-VM
2017-09-25 11:20:16,585 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-378:ctx-d30ac189) Catch Exception: class
com.xensource.xenapi.Types$UuidInvalid due to The uuid you
supplied was invalid.
The uuid you supplied was invalid.
So in my mind I should see that uuid in /dev/mapper/ but I do not.
So I looked at Dag's post and wanted to pull down some info from sql.
+-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
| vmid | vmname | vminstname |
vmdispname | vmacctid | vmacctname | vmdomainid | vmdomname
| vmofferingid | vmspeed | vmmem | volid | volname | volsize |
volpath | voltype | volstate |
+-------+-----------------------------------------+----------------+----------------------+----------+------------------+------------+--------------------+--------------+---------+-------+-------+----------------+--------------+--------------------------------------+----------+----------+
| 223 | DC2FCE | i-23-223-VM | DC2FCE
| 23 | admin | 18 | fce.coop |
29 | 2000 | 4096 | 275 | ROOT-223 | 171798691840 |
f5aabea5-2a18-4f24-9f8d-d750a907a699 | ROOT | Ready |
This shows me that the volpath is *699
Now that matches up with the only VHD symbolic link I can find back to
/dev/mapper/VG_XenStorage--
VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2]# ls *699
VHD-f5aabea5-2a18-4f24-9f8d-d750a907a699
I think the parent volume got deleted and I'm left with only the 699 volume
seen above.
Do you have any suggestions on recovering from this?
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Monday, September 25, 2017 10:15 AM
To: users@cloudstack.apache.org
Subject: RE: Corrupted HDD
Ok so I will be going through both of the items listed.
But my problem is our xenserver is with iSCSI SAN's so we do not have any
VHD files.
# ls
VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--f5aabea5--2a18--4f24--9f8d--d750a907a699
Now I've tried to run a dd on that file to export and try to recover from
that its corrupted.
I'm grasping here and will be looking at more information. I need to
recover data from this VG_XenStorage file.
Thank you in advance. Just wanted to let you know where I am sitting.
Jeremy
-----Original Message-----
From: Dag Sonstebo [mailto:dag.sonst...@shapeblue.com]
Sent: Monday, September 25, 2017 3:16 AM
To: users@cloudstack.apache.org
Subject: Re: Corrupted HDD
Hi Jeremy,
In addition to Adrian’s walk-through – take a look at the following blog
article from last year – it should give you some more options to try -
http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 25/09/2017, 03:49, "Adrian Sender" <asen...@testlabs.com.au> wrote:
Hi Jeremy,
Some time ago I worked with Citrix in creating a article to mount and
recover
LVMoHBA.
How to Mount Linux LVM Partition in a XenServer Host
http://support.citrix.com/article/CTX117791
The above CTX article only applies to raw LVM LV. In our case the VDIs
are LVs
with VHD headers in it. We'll have to use blktap to create tapdisks to
open
the VHD LV before we can create device maps for the partitions / LVs
within
the VDI.
Up to activating the VG, the article still applies. After activating the
VG/LVs, we'll need to manually create tapdisk to open the VHD LV before
using
kpartx to create maps from partition tables.
NOTE: I've included output from my test environment for your reference
1. Create tapdisk
# tap-ctl allocate /dev/xen/blktap-2/tapdev0
2. Spawn a new process
# tap-ctl spawn
tapdisk spawned with pid 30383
# tap-ctl list
30383 - - - -
- 0 - - -
4. Attach the tapdisk to the process
tap-ctl attach -p PID -m minor
# tap-ctl attach -p 30383 -m 0
# tap-ctl list
30383 0 0 - -
5. Open the VHD LV
Note you may need to activate the LVM first with lvchange -ay
# tap-ctl open -p 30383 -m 0 -a
vhd:/dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
6. Verify
# tap-ctl list
30383 0 0 vhd
/dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
7. Create device maps from partition tables using kpartx
kpartx -av /dev/xen/blktap-2/tapdev0
# kpartx -av /dev/xen/blktap-2/tapdev0
add map tapdev0p1 (252:5): 0 1024000 linear /dev/xen/blktap-2/tapdev0
2048
add map tapdev0p2 (252:6): 0 32872448 linear /dev/xen/blktap-2/tapdev0
1026048
NOTE: in this case, it is a CentOS 7 VDI, contains a /boot partition
and VG -
linux and 2 LVs (root, home).
Device maps were created in
/dev/mapper/tapdev0pN
8. Mount 1st partition
# mount /dev/mapper/tapdev0p1 /mnt/boot
9. Activate VG
# vgchange -a y
2 logical volume(s) in volume group "linux" now active
# lvscan
ACTIVE '/dev/linux/root' [13.67 GB] inherit
ACTIVE '/dev/linux/home' [2.00 GB] inherit
10. Mount root and home LVs
mount -t xfs /dev/linux/root /mnt/root
NOTE: XenServer 6.2 kernel does NOT support XFS
mount -t ext4 /dev/linux/home /mnt/home
11. After backing up data, reverse the procedure to release resources
List partition mappings
# kpartx -lv /dev/xen/blktap-2/tapdev0
tapdev0p1 : 0 1024000 /dev/xen/blktap-2/tapdev0 2048
tapdev0p2 : 0 32872448 /dev/xen/blktap-2/tapdev0 1026048
De-active the VG (all LVs)
# vgchange -an linux
0 logical volume(s) in volume group "linux" now active
Delete partition mappings
# kpartx -dv /dev/xen/blktap-2/tapdev0
del devmap : tapdev0p2
del devmap : tapdev0p1
12. Release the blktap resources
# tap-ctl list
30383 0 0 vhd
/dev/VG_XenStorage-b1415cf5-07e2-14fa-cbf8-6490751da242/VHD-182ce043-bf98-45d9-926b-595bbd82ed8a
# tap-ctl close -p 30383 -m 0
[root@xenserver-zilblqhl /]# tap-ctl list
30383 0 0x2 - -
# tap-ctl destroy -p 30383 -m 0
[root@xenserver-zilblqhl /]# tap-ctl list
Regards,
Adrian Sender
-----
dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
----- Original Message -----------
From: Jeremy Peterson <jpeter...@acentek.net>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Sent: Sun, 24 Sep 2017 22:38:29 +0000
Subject: Corrupted HDD
> So I need to know if anyone can drop a hint on this.
>
> How can i recover data from cloudstack with xenserver. One of the
> storage LUN's got 100% full. I was carefully deleting un-associated
> disks from the LUN. A disk that had no VM attached. Then when one
> of the VM's rebooted it came online with no bootable disk found.
>
> Now I'm thinking this over that one of those disks were the a master
> UUID and thats why my VM's wont boot. Can that been fixed?
>
> I look in primary storage and can find the UUID of a disk matches a
> volume in /dev/mapper/VG.....
>
> Can I pull that disk down and create a VHD to deploy a template and
> deploy a new vm from that vhd file?
>
> Can I dd that disk to export to anything to try and recover data?
> Office files or PDF's are critical.
>
> Does anyone know if there are any good consulting companies with
> Citrix or if Citrix can help with this?
>
> Jeremy
------- End of Original Message -------
