Re: [CentOS-virt] Machine freeze
So, I guess this wasn't just a hardware issue. I actually had another system crash. This only appears to happen when I'm issuing xm commands over and over. Any thoughts? Thanks, Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu On Thu, Apr 30, 2009 at 6:27 PM, Mathew S. McCarrell mccar...@gmail.comwrote: I've discovered what the issue is. The machine is rebooting when a sector error occurs on one of the drives that is part of a software RAID where the VMs are currently being stored. Thanks for the help though. Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu On Thu, Apr 30, 2009 at 3:19 PM, Ljubomir Ljubojevic off...@plcomputers.net wrote: Karanbir, can you please, in short, explain to me current status of 64-bit CentOS compared to i386? Is it's maturity same as of i386? I started to actively use CentOS when 4.2 was last version. My decision to use i386-only was based on issues with some (or many?) drivers like madwifi for AR5007, it's unavailability for older PC's, my impression in that time was that it was not stable enough, and the main thing was since I decided to create my own mirror of main and third party repositories for internal use, I went with i386. What is actual gain in using X86_64? Performance in %? Main advantages beside performance? The real question is, does it pay off to spend 20-30 GB of HDD space for X86_64 if i386 does the job nicely? Just a sentence or two would be most appreciated. Karanbir Singh wrote: Mathew S. McCarrell wrote: Well, I'm actually not using a PAExen kernel but I don't believe that I need to be since I'm running the 64-bit version of CentOS. Am I mistaken in that assumption? Matthew, you are right. Also, the idea of running a PAE kernel on CentOS is non relevant ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
Mathew S. McCarrell wrote: Well, I'm actually not using a PAExen kernel but I don't believe that I need to be since I'm running the 64-bit version of CentOS. Am I mistaken in that assumption? Matthew, you are right. Also, the idea of running a PAE kernel on CentOS is non relevant -- Karanbir Singh : http://www.karan.org/ : 2522...@icq ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
Karanbir, can you please, in short, explain to me current status of 64-bit CentOS compared to i386? Is it's maturity same as of i386? I started to actively use CentOS when 4.2 was last version. My decision to use i386-only was based on issues with some (or many?) drivers like madwifi for AR5007, it's unavailability for older PC's, my impression in that time was that it was not stable enough, and the main thing was since I decided to create my own mirror of main and third party repositories for internal use, I went with i386. What is actual gain in using X86_64? Performance in %? Main advantages beside performance? The real question is, does it pay off to spend 20-30 GB of HDD space for X86_64 if i386 does the job nicely? Just a sentence or two would be most appreciated. Karanbir Singh wrote: Mathew S. McCarrell wrote: Well, I'm actually not using a PAExen kernel but I don't believe that I need to be since I'm running the 64-bit version of CentOS. Am I mistaken in that assumption? Matthew, you are right. Also, the idea of running a PAE kernel on CentOS is non relevant ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
I've discovered what the issue is. The machine is rebooting when a sector error occurs on one of the drives that is part of a software RAID where the VMs are currently being stored. Thanks for the help though. Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu On Thu, Apr 30, 2009 at 3:19 PM, Ljubomir Ljubojevic off...@plcomputers.net wrote: Karanbir, can you please, in short, explain to me current status of 64-bit CentOS compared to i386? Is it's maturity same as of i386? I started to actively use CentOS when 4.2 was last version. My decision to use i386-only was based on issues with some (or many?) drivers like madwifi for AR5007, it's unavailability for older PC's, my impression in that time was that it was not stable enough, and the main thing was since I decided to create my own mirror of main and third party repositories for internal use, I went with i386. What is actual gain in using X86_64? Performance in %? Main advantages beside performance? The real question is, does it pay off to spend 20-30 GB of HDD space for X86_64 if i386 does the job nicely? Just a sentence or two would be most appreciated. Karanbir Singh wrote: Mathew S. McCarrell wrote: Well, I'm actually not using a PAExen kernel but I don't believe that I need to be since I'm running the 64-bit version of CentOS. Am I mistaken in that assumption? Matthew, you are right. Also, the idea of running a PAE kernel on CentOS is non relevant ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
Hey, I'm wondering if it is possible that your problem is related to mine. Earlier today I had to restart one of our domUs on one of our systems. I used xm shutdown instead of xm destroy and then did xm list to determine if the domU had shutdown or not. Upon issuing xm list a second time, the entire server crashed and rebooted. I've checked the logs and have yet to find anything. I've attached a transcript of the commands as I executed them on the server. The system is running CentOS 5.3 x64 w/Xen (kernel 2.6.18-128.1.6.el5xen). Any thoughts? Thanks, Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu 2009/4/7 Maros Timko tim...@gmail.com Hi all, thanks to all for valuable replies. It seems like we identified the issue. We assured that it is not HW related as it was already reproduced on different machines and platforms, with different BIOS versions. We are running a system performance/statistics collector that executes xentop command on Dom0 regularly. This is causing issues. If we execute: xentop -b -d 0.1 /dev/null in multiple instances, it will freeze the system. It was reproduced on CentOS 5.3 (kernel-xen-2.6.18-128.1.6.el5) system. There is created a bug for this issue: http://bugs.centos.org/view.php?id=3454 With regards, Tino 2009/4/3 Maros TIMKO ti...@pobox.sk Hi all, we are running CentOS 5.2 Xen virtualization system with the latest CentOS packages with couple of VMs on DELL PowerEdge. Sometimes the whole machine freezes without anything in log files, anything on the console. Sometimes really means we cannot define why or when. Sometimes the machine was idle with just one VM, sometimes quite busy with couple of VMs. Has anybody had the same experience? If yes, any hints on how to resolve it or how to trace the cause? Thanks. ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt [mccar...@isengard ~]$ ssh xen1 ___ __ _ ___ / \ \ / -_) _ \/ / /_\_\\__/_//_/_/ Last login: Mon Apr 27 11:26:22 2009 from isengard.cslabs.clarkson.edu [mccar...@xen1 ~]$ sudo xm list We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. Password: Sorry, try again. Password: Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 9899 8 r- 12458.2 atp 11 255 1 -b 72748.8 auth 1 127 1 -b121.7 autoguilt 2 255 1 -b523.2 dukr 3 255 1 -b770.4 list 5 255 1 -b191.4 management 6 255 1 -b517.4 osp1 7 255 1 -b 70.4 osp2 8 255 1 -b 68.8 tremulous 9 255 1 -b 397287.4 [mccar...@xen1 ~]$ xm console atp ERROR Internal error: Could not obtain handle on privileged command interface (13 = Permission denied) Error: Most commands need root access. Please try again as root. [mccar...@xen1 ~]$ sudo xm console atp Out of Memory: Kill process 2626 (TreeLimitedRun) score 96585 and children. Out of memory: Killed process 2627 (spectrum). Out of Memory: Kill process 2694 (TreeLimitedRun) score 96565 and children. Out of memory: Killed process 2695 (spectrum). Out of Memory: Kill process 2914 (TreeLimitedRun) score 96210 and children. Out of memory: Killed process 2915 (spectrum). Out of Memory: Kill process 3014 (TreeLimitedRun) score 96153 and children. Out of memory: Killed process 3015 (spectrum). Out of Memory: Kill process 3018 (TreeLimitedRun) score 96177 and children. Out of memory: Killed process 3019 (spectrum). Out of Memory: Kill process 4466 (spectrum) score 189626 and children. Out of memory: Killed process 4466 (spectrum). Out of Memory: Kill process 6324 (TreeLimitedRun) score 96129 and children. Out of memory: Killed process 6325 (spectrum). Out of Memory: Kill process 10680 (TreeLimitedRun) score 96147 and children. Out of memory: Killed process 10681 (spectrum). Out of Memory: Kill process 10800 (TreeLimitedRun) score 96167 and children. Out of memory: Killed process 10801 (spectrum). Out of Memory: Kill process 10852 (TreeLimitedRun) score 96159 and children. Out of memory: Killed process 10853
Re: [CentOS-virt] Machine freeze
Hi Mathew, I would say no. Our system has freezed completely, it did not reboot. Our issue was caused by concurrent access to scheduler method that created a deadlock.I can see some out of memory messages, do you still have enough memory for Dom0? 2009/4/29 Mathew S. McCarrell mccar...@gmail.com Hey, I'm wondering if it is possible that your problem is related to mine. Earlier today I had to restart one of our domUs on one of our systems. I used xm shutdown instead of xm destroy and then did xm list to determine if the domU had shutdown or not. Upon issuing xm list a second time, the entire server crashed and rebooted. I've checked the logs and have yet to find anything. I've attached a transcript of the commands as I executed them on the server. The system is running CentOS 5.3 x64 w/Xen (kernel 2.6.18-128.1.6.el5xen). Any thoughts? Thanks, Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu 2009/4/7 Maros Timko tim...@gmail.com Hi all, thanks to all for valuable replies. It seems like we identified the issue. We assured that it is not HW related as it was already reproduced on different machines and platforms, with different BIOS versions. We are running a system performance/statistics collector that executes xentop command on Dom0 regularly. This is causing issues. If we execute: xentop -b -d 0.1 /dev/null in multiple instances, it will freeze the system. It was reproduced on CentOS 5.3 (kernel-xen-2.6.18-128.1.6.el5) system. There is created a bug for this issue: http://bugs.centos.org/view.php?id=3454 With regards, Tino 2009/4/3 Maros TIMKO ti...@pobox.sk Hi all, we are running CentOS 5.2 Xen virtualization system with the latest CentOS packages with couple of VMs on DELL PowerEdge. Sometimes the whole machine freezes without anything in log files, anything on the console. Sometimes really means we cannot define why or when. Sometimes the machine was idle with just one VM, sometimes quite busy with couple of VMs. Has anybody had the same experience? If yes, any hints on how to resolve it or how to trace the cause? Thanks. ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
Yeah, the Dom0 should have plenty of memory left since only 2-3 GB of memory is being used out of 12 GB installed. The out of memory messages were from the domU that I xm consoled into prior to shutting down that particular VM because it was out of memory. Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu On Wed, Apr 29, 2009 at 3:09 PM, Maros Timko tim...@gmail.com wrote: Hi Mathew, I would say no. Our system has freezed completely, it did not reboot. Our issue was caused by concurrent access to scheduler method that created a deadlock.I can see some out of memory messages, do you still have enough memory for Dom0? 2009/4/29 Mathew S. McCarrell mccar...@gmail.com Hey, I'm wondering if it is possible that your problem is related to mine. Earlier today I had to restart one of our domUs on one of our systems. I used xm shutdown instead of xm destroy and then did xm list to determine if the domU had shutdown or not. Upon issuing xm list a second time, the entire server crashed and rebooted. I've checked the logs and have yet to find anything. I've attached a transcript of the commands as I executed them on the server. The system is running CentOS 5.3 x64 w/Xen (kernel 2.6.18-128.1.6.el5xen). Any thoughts? Thanks, Matt -- Mathew S. McCarrell Clarkson University '10 mccar...@gmail.com mccar...@clarkson.edu 2009/4/7 Maros Timko tim...@gmail.com Hi all, thanks to all for valuable replies. It seems like we identified the issue. We assured that it is not HW related as it was already reproduced on different machines and platforms, with different BIOS versions. We are running a system performance/statistics collector that executes xentop command on Dom0 regularly. This is causing issues. If we execute: xentop -b -d 0.1 /dev/null in multiple instances, it will freeze the system. It was reproduced on CentOS 5.3 (kernel-xen-2.6.18-128.1.6.el5) system. There is created a bug for this issue: http://bugs.centos.org/view.php?id=3454 With regards, Tino 2009/4/3 Maros TIMKO ti...@pobox.sk Hi all, we are running CentOS 5.2 Xen virtualization system with the latest CentOS packages with couple of VMs on DELL PowerEdge. Sometimes the whole machine freezes without anything in log files, anything on the console. Sometimes really means we cannot define why or when. Sometimes the machine was idle with just one VM, sometimes quite busy with couple of VMs. Has anybody had the same experience? If yes, any hints on how to resolve it or how to trace the cause? Thanks. ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Machine freeze
Yes, mem and disk check was also our first thing to do. But it happened on different machines (1950s and 2950s), different BIOS versions and number of NICs. The freeze situation is unrecoverable - machine replies to pings, but did not write anything to console. You cannot SSH to it, the only thing we could do is power down the machine. After that everything is fine. Thanks. 2009/4/3 Tim Verhoeven tim.verhoeven...@gmail.com On Fri, Apr 3, 2009 at 4:18 PM, Maros TIMKO ti...@pobox.sk wrote: we are running CentOS 5.2 Xen virtualization system with the latest CentOS packages with couple of VMs on DELL PowerEdge. Sometimes the whole machine freezes without anything in log files, anything on the console. Sometimes really means we cannot define why or when. Sometimes the machine was idle with just one VM, sometimes quite busy with couple of VMs. Has anybody had the same experience? If yes, any hints on how to resolve it or how to trace the cause? The complete freezing of a machine like that sounds like a hardware issue to me, most likely the memory. Does the machine unfreeze after a while or do you have to power cycle the server when it happens ? I would suggest running a memtest. Regards, Tim -- Tim Verhoeven - tim.verhoeven...@gmail.com - 0479 / 88 11 83 Hoping the problem magically goes away by ignoring it is the microsoft approach to programming and should never be allowed. (Linus Torvalds) ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
