Yep, absolutely. Unfortunately, probably 99% of the 'omg my VM's on my server are down' calls I get are from the folks who don't understand the platform like you or I. So someone increasing a VM from 16 to to 30GB of ram on a server with 32GB of physical ram, and then wondering why they are now getting out of disk space errors when they had 30GB free before modifying the VM, is a common call.
And yeah, thin provisioning, argh! On 21 November 2017 at 07:51, Robert Hudson <hud...@gmail.com> wrote: > > > On 20 Nov. 2017 10:20 pm, "Damien Gardner Jnr" <rend...@rendrag.net> > wrote: > >> I noticed he was talking Veeam - I've often seen a volume run out of >> space while Veeam was trying to sync an off-site replica, and then the >> snapshots get stuck. Same if Veeam has crashed or network has dropped, >> while doing the sync - the snapshot never gets cleaned up. >> > > Yes, various backup apps that interact with the VMFS file system via the > APIs can leave things lying around - but you need to know this, watch out > for it and act appropriately. > > He also mentioned Swapfiles. That reeks of someone not reserving memory >> for the VM. Would be worth checking that in the settings. I know there >> are valid reasons for over-committing ram on VM's compared to the >> hypervisor's physical memory, but unless you know exactly what you're >> doing, and are keeping a watch on it, it's easy to totally fill a datastore >> and get yourself into trouble. If you have the Physical ram to provide all >> your VM's needs, it's better to just reserve that memory in the settings, >> and then you don't need all that extra disk space. >> > > Memory reservations are not necessary (or even considered best practice) > for the vast majority of workloads. Sure, reserve memory for VMs running > Java within them (as the hypervisor can't see what the JVM's memory > management is doing), and for specific apps like Oracle and SQL database > servers, but in general, there's really no need, nor substantial benefit. > > The swapfile overhead is capped at the virtual RAM assigned per VM. It > really isn't hard to manage. Sure, the swapfiles disappear when the VM is > shut down, but they come back again when the VMs are booted (and the VMs > won't boot if you don't have space for the swap files). > > If a VMFS (or NFS for that matter) file system runs out of space, the VMs > will freeze. This is only easy if the environment is not appropriately > managed. Snapshots consume an increasing amount of space as they live > longer (whereas swapfiles do not), and are a key monitoring point in a > virtualisation environment. > > Another key monitoring point is thin provisioning - it's great telling > your guest OS it has a 100GB system drive, but only writing ~20GB in the > VMFS volume when you install your OS. But you then need to realise that > your thin provisioned VMDK *will* grow as new data is written within the > VM. And it will grow to up to 100GB, depending on how much data the VM > writes. > > Monitoring (and alerting, and then acting on the alerts received) is key > to running a virtualisation environment successfully, on anything from a > single small host with a couple of VMs to a massive environment with > hundreds of hosts and thousands of VMs. There are more monitoring point > than are helpful in the vast majority of situations, but then there are > good free tools available that can filter down the noise into usable data > (and some of them not only point out important issues, but then also tell > you what to do about them). > > What it reeks of is not understanding the environment and toolset > properly, and making mistakes accordingly. The good thing about this is > that it is easily remedied though education (self or instructor-led) and > then applying what is learned to the environment. > > If you don't know what you're doing, you shouldn't be touching > virtualisation. Yep, that's seemingly harsh - but it's absolutely fair - > the repercussions otherwise can be dire. > > >> On 20 November 2017 at 15:47, Robert Hudson <hud...@gmail.com> wrote: >> >>> VMs only have a snapshot when something had told VMware (ESXi or >>> vCenter, which tells ESXi) to create a snapshot. They only have large >>> snapshots or chains of snapshots if those snapshots aren't being cleaned up. >>> >>> Snapshots in this case are completely independent of the guest OS, and >>> have nothing to do with where files within those guest OSs may be being >>> written to within the guest OS fike system. >>> >>> On 20 Nov. 2017 1:08 pm, "Rory Jones" <rory.jones...@gmail.com> wrote: >>> >>> As others have said, I wouldn’t go messing around with those VMware >>> files. You have a very high chance of borking something. Instead, get to >>> the source of the problem. Find out *why* your VMs have a sudden appetite >>> for data. Unfortunately I can’t speak for Linux, but on the off chance the >>> suspect VMs are running Windows, there’s an awesome little tool called >>> WinDirStat. For the record, for Mac the equivalent is Disk Inventory X. Run >>> these tools, or the equivalent for your OS. It’ll give you a pretty picture >>> like you’ve just broken Tetris of the contents of the drive. Go for the big >>> blocks first. Find out what they are. Diagnose from there. >>> >>> Hope this helps someone in the future. >>> >>> Kind regards, >>> Rory >>> ------------------------------ >>> *From:* AusNOG <ausnog-boun...@lists.ausnog.net> on behalf of Daniel >>> Watson <dgwatson1...@gmail.com> >>> *Sent:* Monday, November 20, 2017 8:03:19 AM >>> *To:* Burt Mascareigne >>> *Cc:* ausnog@lists.ausnog.net >>> >>> *Subject:* Re: [AusNOG] VMWare Snapshot / Delta VMDK removal >>> >>> Have a QNAP. So moved the pagefile’s to an ISCIS target. And then >>> removed all the Delta’s >>> >>> Freed up 268gb which has allowed the Veeam backups to run >>> >>> Daniel >>> >>> Sent from my iPhone >>> >>> > On 20 Nov 2017, at 8:09 am, Burt Mascareigne <b...@stormnetwork.com.au> >>> wrote: >>> > >>> > I feel like you dodged a bullet there. >>> > >>> > If you were not able to shut it down, then get a NAS or some device >>> and clone to it at least that will give you a slimmed down clean version. >>> > >>> > >>> > >>> > Regards, >>> > >>> > >>> > Burt Mascareigne >>> > Mobile 0414 450 962 Office (02) 9965 5422 >>> > Address Level 19, 1 O’Connell Street, Sydney NSW 2000 >>> <https://maps.google.com/?q=Level+19,+1+O%E2%80%99Connell+Street,+Sydney+NSW+2000&entry=gmail&source=g> >>> > Web http://www.stormnetwork.com.au >>> > >>> > >>> > >>> > -----Original Message----- >>> > From: AusNOG [mailto:ausnog-boun...@lists.ausnog.net >>> <ausnog-boun...@lists.ausnog.net>] On Behalf Of Daniel Watson >>> > Sent: Sunday, 19 November 2017 10:23 PM >>> > To: ausnog@lists.ausnog.net >>> > Subject: Re: [AusNOG] VMWare Snapshot / Delta VMDK removal >>> > >>> > Thanks to all those fantastic people who have responded >>> > >>> > I have powered off this particular VM for the time being. And have >>> recovered 35gb in swapfile >>> > >>> > I have taken a new successful snapshot. Which then allowed me to >>> delete “all” snapshots. Including those old delta files. This will take a >>> few hours to complete. But then should give me back 100gb of space >>> > >>> > Appreciate all the assistance. I’ll keep you posted how it goes >>> tomorrow >>> > >>> > Daniel >>> > >>> > Sent from my iPhone >>> > >>> >> On 19 Nov 2017, at 9:47 pm, Daniel Watson <dgwatson1...@gmail.com> >>> wrote: >>> >> >>> >> Sorry to bother the list but this is critical >>> >> >>> >> Wanted to know from others. If it is safe to remove a delta VMDK file >>> from a running vm? >>> >> >>> >> A server I am now looking after is running out of space like no >>> tomorrow. And there are 3 delta VMDK ‘s taking up 80gb of space >>> >> >>> >> Any information would be appreciated >>> >> >>> >> Daniel >>> >> >>> >> Sent from my iPhone >>> >> _______________________________________________ >>> >> AusNOG mailing list >>> >> AusNOG@lists.ausnog.net >>> >> http://lists.ausnog.net/mailman/listinfo/ausnog >>> > _______________________________________________ >>> > AusNOG mailing list >>> > AusNOG@lists.ausnog.net >>> > http://lists.ausnog.net/mailman/listinfo/ausnog >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG@lists.ausnog.net >>> http://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG@lists.ausnog.net >>> http://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >>> >>> _______________________________________________ >>> AusNOG mailing list >>> AusNOG@lists.ausnog.net >>> http://lists.ausnog.net/mailman/listinfo/ausnog >>> >>> >> >> >> -- >> >> Damien Gardner Jnr >> VK2TDG. Dip EE. GradIEAust >> rend...@rendrag.net - http://www.rendrag.net/ >> -- >> We rode on the winds of the rising storm, >> We ran to the sounds of thunder. >> We danced among the lightning bolts, >> and tore the world asunder >> > -- Damien Gardner Jnr VK2TDG. Dip EE. GradIEAust rend...@rendrag.net - http://www.rendrag.net/ -- We rode on the winds of the rising storm, We ran to the sounds of thunder. We danced among the lightning bolts, and tore the world asunder
_______________________________________________ AusNOG mailing list AusNOG@lists.ausnog.net http://lists.ausnog.net/mailman/listinfo/ausnog
