Re: [CentOS-virt] Missing packages

2010-07-13 Thread Mathew S. McCarrell
On Tue, Jul 13, 2010 at 2:22 PM, David Martin  wrote:
>
> What about something like oo.org that isn't part of server errata but
> still gets updated.
>
>
That's a good point... I guess I'm not entirely sure about all of the errata
feeds that will show which updates make it into CentOS.  Perhaps someone
from the CentOS team can give you a better answer.

Matt

--
Mathew S. McCarrell
Clarkson University '10 '11

mccar...@gmail.com
mccar...@clarkson.edu
1-518-314-9214
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Missing packages

2010-07-13 Thread Mathew S. McCarrell
On Tue, Jul 13, 2010 at 1:53 PM, David Martin  wrote:

> I haven't seen anything on the announce list and don't see these in the
> repos:
>
> qspice-client - http://rhn.redhat.com/errata/RHEA-2010-0492.html
> spice-xpi - http://rhn.redhat.com/errata/RHEA-2010-0493.html
> kvm - http://rhn.redhat.com/errata/RHBA-2010-0479.html
>
> Anyone know what the story is?
>
>
You are not seeing those because they don't apply to RHEL Server.

See http://rhn.redhat.com/errata/rhel-server-errata.html for the correct
updates.

HTH,
Matt

--
Mathew S. McCarrell
Clarkson University '10 '11

mccar...@gmail.com
mccar...@clarkson.edu
1-518-314-9214
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Logrotate/cron and major I/O contention with KVM.

2010-03-11 Thread Mathew S. McCarrell
On Wed, Mar 10, 2010 at 5:28 PM, Steven Ellis wrote:

>
> Is anyone else having major I/O peaks due to logrotate or other jobs
> running simultaneously across multiple guests. I have one KVM server running
> Centos 5.4 with local disk that is seriously suffering as most of the guests
> rotate their syslog at the same time.
>
> Looking at the KVM server I'm seeing
>
> 11:00:01 PM   CPU %user %nice   %system   %iowait%steal
> %idle
> 03:40:01 AM   all  0.07  0.00  2.74  0.93  0.00
> 96.26
> 03:50:01 AM   all  0.07  0.00  1.17  1.18  0.00
> 97.58
> 04:00:01 AM   all  0.08  0.00  1.51  0.82  0.00
> 97.59
> 04:10:02 AM   all  0.53  0.03 15.31 51.61  0.00
> 32.53
> 04:20:01 AM   all  0.28  0.12  4.12 22.21  0.00
> 73.27
> 04:30:01 AM   all  0.07  0.00  0.80  1.21  0.00
> 97.92
> 04:40:01 AM   all  0.07  0.00  2.60  1.81  0.00
> 95.52
> 04:50:01 AM   all  0.08  0.00  0.79  1.44  0.00
> 97.69
>
> On one of the guests running Centos 4.6 the impact is so bad I get DMA
> timeout errors in the syslog, and occasional kernel panics.
>
> Mar 11 04:05:04 localhost kernel: hda: dma_timer_expiry: dma status == 0x21
> Mar 11 04:05:14 localhost kernel: hda: DMA timeout error
> Mar 11 04:05:14 localhost kernel: hda: dma timeout error: status=0x50 {
> DriveReady SeekComplete }
> Mar 11 04:05:14 localhost kernel:
> Mar 11 04:05:14 localhost kernel: ide: failed opcode was: unknown
> Mar 11 04:05:59 localhost kernel: hda: dma_timer_expiry: dma status == 0x21
> Mar 11 04:06:14 localhost kernel: hda: DMA timeout error
> Mar 11 04:06:14 localhost kernel: hda: dma timeout error: status=0x50 {
> DriveReady SeekComplete }
>
>
> One reference I've found is at
>  * http://lonesysadmin.net/linux-virtual-machine-tuning-guide/
>
> This suggests avoiding running scheduled jobs simultaneously across guests,
> and suggests using a random sleep.
>
> Does anyone else have suggestions on reducing the impact of cron/logrotate.
>
>
I ran into this issue as well on a box running Xen with local storage.

My solution was to modify /etc/crontab to run /etc/cron.weekly at different
times for each guest and for the dom0.  I modified the entry on each VM to
be 10 minutes after the previous one and have not seen any load spikes since
then.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu
1-518-314-9214
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Tiny OT: yum priorities or protectbase

2009-06-29 Thread Mathew S. McCarrell
Personally, I would go with priorities because it gives you more control.

I'm currently in the process of switching to priorities because of packages
from external repos replacing the stable ones that come with CentOS even
though I was using protectbase.  I'm not entirely sure this is what was
causing my reboot issues on the Xen boxes that I run but it certainly could
have been a factor.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Mon, Jun 29, 2009 at 5:12 PM, Ben Montanelli wrote:

> I'm elevating myself from "noob" to "noob with scars." I'm not one to
> ask a lot of questions if I have time to find out on my own.
>
> I now know at least a half dozen ways to hose a solid CentOS Xen
> install. That's a good thing, I learn from mistakes. Like forgetting to
> actually completely configure priority statements in the repo confs.
>
> In the past I have used yum protect-base, but about a year ago or so I
> switched to yum priorities because it seems a bit more granular.
>
> Is there a recommended preference on CentOS Xen installs? I'm pretty
> conservative now and won't go near Plus or Contribs, I do have Convirt
> repos in and it looks like they pulldown a couple dependencies from
> RPMForge (paramiko and socat).
>
> I don't plan on anymore repos, but am wary enough to want to nail my
> last install down.
>
> Any prefs here for CentOS Xen? Yum Protect-base or Priorities?
>
>
> ___
> CentOS-virt mailing list
> CentOS-virt@centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Bad Disk Performance of domU

2009-05-20 Thread Mathew S. McCarrell
In general, disk performance on a domU will always be lower than on a dom0.
That being said, I agree with Christopher that is sounds like you are using
a sparse disk file.

Did you create this VM with virt-install?  If so, you can pass the
--nonsparse option to virt-install when creating the VM to allocate all the
disk space when creating the VM.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Wed, May 20, 2009 at 12:35 PM, Francisco Pérez wrote:

> LVM on dom0 or domu?
>
> 2009/5/20 Christopher G. Stach II 
>
> - "Francisco Pérez"  wrote:
>>
>> > I guess the bad performance it goes with the writing. For example on
>> > decompression of a big tar.gz, installing rpm packages. Also on
>> > restoring a MySQL Database...
>> >
>> > The first clue when i knew that something were wrong was on the FS
>> > formating stage. It took really long time to format 40 GB to ext3. I
>> > did the same on other machine with a sata disk... surprise! same
>> > software and hardware config except for the inferior disk type and did
>> > it in no time.
>>
>> It sounds like your the virtual disk file is sparse.  If that's the case,
>> you traded performance for space.  It not only has to deal with the
>> underlying filesystem, but it allocates the space on demand and hits that
>> filesystem's journal.  Try an LVM logical disk and see how that performs.
>>
>> --
>> Christopher G. Stach II
>>
>>
>> ___
>> CentOS-virt mailing list
>> CentOS-virt@centos.org
>> http://lists.centos.org/mailman/listinfo/centos-virt
>>
>
>
> ___
> CentOS-virt mailing list
> CentOS-virt@centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
>
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Machine freeze

2009-05-05 Thread Mathew S. McCarrell
So, I guess this wasn't just a hardware issue.  I actually had another
system crash.

This only appears to happen when I'm issuing xm commands over and over.

Any thoughts?

Thanks,
Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Thu, Apr 30, 2009 at 6:27 PM, Mathew S. McCarrell wrote:

> I've discovered what the issue is.
>
> The machine is rebooting when a sector error occurs on one of the drives
> that is part of a software RAID where the VMs are currently being stored.
>
> Thanks for the help though.
>
> Matt
>
> --
> Mathew S. McCarrell
> Clarkson University '10
>
> mccar...@gmail.com
> mccar...@clarkson.edu
>
>
> On Thu, Apr 30, 2009 at 3:19 PM, Ljubomir Ljubojevic <
> off...@plcomputers.net> wrote:
>
>> Karanbir, can you please, in short, explain to me current status of
>> 64-bit CentOS compared to i386? Is it's maturity same as of i386?
>>
>> I started to actively use CentOS when 4.2 was last version. My decision
>> to use i386-only was based on issues with some (or many?) drivers like
>> madwifi for AR5007, it's unavailability for older PC's, my impression in
>> that time was that it was not stable enough, and the main thing was
>> since I decided to create my own mirror of main and third party
>> repositories for internal use, I went with i386.
>>
>> What is actual gain in using X86_64? Performance in %? Main advantages
>> beside performance? The real question is, does it pay off to spend 20-30
>> GB of HDD space for X86_64 if i386 does the job nicely? Just a sentence
>> or two would be most appreciated.
>>
>> Karanbir Singh wrote:
>> > Mathew S. McCarrell wrote:
>> >> Well, I'm actually not using a PAExen kernel but I don't believe that I
>> >> need to be since I'm running the 64-bit version of CentOS.  Am I
>> >> mistaken in that assumption?
>> >>
>> >
>> > Matthew, you are right.
>> >
>> > Also, the idea of running a PAE kernel on CentOS is non relevant
>> >
>>
>> ___
>> CentOS-virt mailing list
>> CentOS-virt@centos.org
>> http://lists.centos.org/mailman/listinfo/centos-virt
>>
>
>
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Machine freeze

2009-04-30 Thread Mathew S. McCarrell
I've discovered what the issue is.

The machine is rebooting when a sector error occurs on one of the drives
that is part of a software RAID where the VMs are currently being stored.

Thanks for the help though.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Thu, Apr 30, 2009 at 3:19 PM, Ljubomir Ljubojevic  wrote:

> Karanbir, can you please, in short, explain to me current status of
> 64-bit CentOS compared to i386? Is it's maturity same as of i386?
>
> I started to actively use CentOS when 4.2 was last version. My decision
> to use i386-only was based on issues with some (or many?) drivers like
> madwifi for AR5007, it's unavailability for older PC's, my impression in
> that time was that it was not stable enough, and the main thing was
> since I decided to create my own mirror of main and third party
> repositories for internal use, I went with i386.
>
> What is actual gain in using X86_64? Performance in %? Main advantages
> beside performance? The real question is, does it pay off to spend 20-30
> GB of HDD space for X86_64 if i386 does the job nicely? Just a sentence
> or two would be most appreciated.
>
> Karanbir Singh wrote:
> > Mathew S. McCarrell wrote:
> >> Well, I'm actually not using a PAExen kernel but I don't believe that I
> >> need to be since I'm running the 64-bit version of CentOS.  Am I
> >> mistaken in that assumption?
> >>
> >
> > Matthew, you are right.
> >
> > Also, the idea of running a PAE kernel on CentOS is non relevant
> >
>
> ___
> CentOS-virt mailing list
> CentOS-virt@centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Machine freeze

2009-04-29 Thread Mathew S. McCarrell
Well, I'm actually not using a PAExen kernel but I don't believe that I need
to be since I'm running the 64-bit version of CentOS.  Am I mistaken in that
assumption?

Thanks,
Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Wed, Apr 29, 2009 at 5:06 PM, Ljubomir Ljubojevic  wrote:

> Just to make sure, you are using PAExen kernel on 12GB RAM Linux? 4GB is
>  max for regular kernel without PAE extensions.
> Mathew S. McCarrell wrote:
> > Yeah, the Dom0 should have plenty of memory left since only 2-3 GB of
> > memory is being used out of 12 GB installed.  The out of memory messages
> > were from the domU that I xm consoled into prior to shutting down that
> > particular VM because it was out of memory.
> >
> > Matt
> >
> > --
> > Mathew S. McCarrell
> > Clarkson University '10
> >
> > mccar...@gmail.com <mailto:mccar...@gmail.com>
> > mccar...@clarkson.edu <mailto:mccar...@clarkson.edu>
> >
> >
> > On Wed, Apr 29, 2009 at 3:09 PM, Maros Timko  > <mailto:tim...@gmail.com>> wrote:
> >
> > Hi Mathew,
> >
> > I would say no. Our system has freezed completely, it did not
> > reboot. Our issue was caused by concurrent access to scheduler
> > method that created a deadlock.
> > I can see some out of memory messages, do you still have enough
> > memory for Dom0?
> >
> > 2009/4/29 Mathew S. McCarrell  > <mailto:mccar...@gmail.com>>
> >
> > Hey,
> >
> > I'm wondering if it is possible that your problem is related to
> > mine.  Earlier today I had to restart one of our domUs on one of
> > our systems.  I used xm shutdown instead of xm destroy and then
> > did xm list to determine if the domU had shutdown or not.  Upon
> > issuing xm list a second time, the entire server crashed and
> > rebooted.
> >
> > I've checked the logs and have yet to find anything.  I've
> > attached a transcript of the commands as I executed them on the
> > server.  The system is running CentOS 5.3 x64 w/Xen (kernel
> > 2.6.18-128.1.6.el5xen).
> >
> > Any thoughts?
> >
> > Thanks,
> > Matt
> >
> > --
> > Mathew S. McCarrell
> > Clarkson University '10
> >
> > mccar...@gmail.com <mailto:mccar...@gmail.com>
> > mccar...@clarkson.edu <mailto:mccar...@clarkson.edu>
> >
> >
> > 2009/4/7 Maros Timko mailto:tim...@gmail.com
> >>
> >
> > Hi all,
> >
> > thanks to all for valuable replies.
> > It seems like we identified the issue. We assured that it is
> > not HW related as it was already reproduced on different
> > machines and platforms, with different BIOS versions.
> > We are running a system performance/statistics collector
> > that executes "xentop" command on Dom0 regularly. This is
> > causing issues. If we execute:
> > xentop -b -d 0.1 > /dev/null
> > in multiple instances, it will freeze the system.
> > It was reproduced on CentOS 5.3
> > (kernel-xen-2.6.18-128.1.6.el5) system.
> > There is created a bug for this issue:
> > http://bugs.centos.org/view.php?id=3454
> >
> > With regards,
> >
> > Tino
> >
> >
> > 2009/4/3 Maros TIMKO mailto:ti...@pobox.sk
> >>
> >
> > Hi all,
> >
> > we are running CentOS 5.2 Xen virtualization system with
> > the latest CentOS packages with couple of VMs on DELL
> > PowerEdge. "Sometimes" the whole machine freezes without
> > anything in log files, anything on the console.
> > "Sometimes" really means we cannot define why or when.
> > Sometimes the machine was idle with just one VM,
> > sometimes quite busy with couple of VMs.
> >
> >  Has anybody had the same experience? If yes, any hints
> > on how to resolve it or how to trace the cause?
> >
> >
> >
> > Thanks.
> >
> >
> > ___
> > CentOS

Re: [CentOS-virt] Machine freeze

2009-04-29 Thread Mathew S. McCarrell
Yeah, the Dom0 should have plenty of memory left since only 2-3 GB of memory
is being used out of 12 GB installed.  The out of memory messages were from
the domU that I xm consoled into prior to shutting down that particular VM
because it was out of memory.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


On Wed, Apr 29, 2009 at 3:09 PM, Maros Timko  wrote:

> Hi Mathew,
>
> I would say no. Our system has freezed completely, it did not reboot. Our
> issue was caused by concurrent access to scheduler method that created a
> deadlock.I can see some out of memory messages, do you still have enough
> memory for Dom0?
>
> 2009/4/29 Mathew S. McCarrell 
>
> Hey,
>>
>> I'm wondering if it is possible that your problem is related to mine.
>> Earlier today I had to restart one of our domUs on one of our systems.  I
>> used xm shutdown instead of xm destroy and then did xm list to determine if
>> the domU had shutdown or not.  Upon issuing xm list a second time, the
>> entire server crashed and rebooted.
>>
>> I've checked the logs and have yet to find anything.  I've attached a
>> transcript of the commands as I executed them on the server.  The system is
>> running CentOS 5.3 x64 w/Xen (kernel 2.6.18-128.1.6.el5xen).
>>
>> Any thoughts?
>>
>> Thanks,
>> Matt
>>
>> --
>> Mathew S. McCarrell
>> Clarkson University '10
>>
>> mccar...@gmail.com
>> mccar...@clarkson.edu
>>
>>
>> 2009/4/7 Maros Timko 
>>
>> Hi all,
>>>
>>> thanks to all for valuable replies.
>>> It seems like we identified the issue. We assured that it is not HW
>>> related as it was already reproduced on different machines and platforms,
>>> with different BIOS versions.
>>> We are running a system performance/statistics collector that executes
>>> "xentop" command on Dom0 regularly. This is causing issues. If we execute:
>>> xentop -b -d 0.1 > /dev/null
>>> in multiple instances, it will freeze the system.
>>> It was reproduced on CentOS 5.3 (kernel-xen-2.6.18-128.1.6.el5) system.
>>> There is created a bug for this issue:
>>> http://bugs.centos.org/view.php?id=3454
>>>
>>> With regards,
>>>
>>> Tino
>>>
>>>
>>> 2009/4/3 Maros TIMKO 
>>>
>>>> Hi all,
>>>>
>>>> we are running CentOS 5.2 Xen virtualization system with the latest
>>>> CentOS packages with couple of VMs on DELL PowerEdge. "Sometimes" the whole
>>>> machine freezes without anything in log files, anything on the console.
>>>> "Sometimes" really means we cannot define why or when. Sometimes the 
>>>> machine
>>>> was idle with just one VM, sometimes quite busy with couple of VMs.
>>>>
>>>>  Has anybody had the same experience? If yes, any hints on how to
>>>> resolve it or how to trace the cause?
>>>>
>>>>
>>>>
>>>> Thanks.
>>>>
>>>> ___
>>>> CentOS-virt mailing list
>>>> CentOS-virt@centos.org
>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>
>>>>
>>>
>>> ___
>>> CentOS-virt mailing list
>>> CentOS-virt@centos.org
>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>
>>>
>>
>> ___
>> CentOS-virt mailing list
>> CentOS-virt@centos.org
>> http://lists.centos.org/mailman/listinfo/centos-virt
>>
>>
>
> ___
> CentOS-virt mailing list
> CentOS-virt@centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
>
___
CentOS-virt mailing list
CentOS-virt@centos.org
http://lists.centos.org/mailman/listinfo/centos-virt


Re: [CentOS-virt] Machine freeze

2009-04-29 Thread Mathew S. McCarrell
Hey,

I'm wondering if it is possible that your problem is related to mine.
Earlier today I had to restart one of our domUs on one of our systems.  I
used xm shutdown instead of xm destroy and then did xm list to determine if
the domU had shutdown or not.  Upon issuing xm list a second time, the
entire server crashed and rebooted.

I've checked the logs and have yet to find anything.  I've attached a
transcript of the commands as I executed them on the server.  The system is
running CentOS 5.3 x64 w/Xen (kernel 2.6.18-128.1.6.el5xen).

Any thoughts?

Thanks,
Matt

--
Mathew S. McCarrell
Clarkson University '10

mccar...@gmail.com
mccar...@clarkson.edu


2009/4/7 Maros Timko 

> Hi all,
>
> thanks to all for valuable replies.
> It seems like we identified the issue. We assured that it is not HW related
> as it was already reproduced on different machines and platforms, with
> different BIOS versions.
> We are running a system performance/statistics collector that executes
> "xentop" command on Dom0 regularly. This is causing issues. If we execute:
> xentop -b -d 0.1 > /dev/null
> in multiple instances, it will freeze the system.
> It was reproduced on CentOS 5.3 (kernel-xen-2.6.18-128.1.6.el5) system.
> There is created a bug for this issue:
> http://bugs.centos.org/view.php?id=3454
>
> With regards,
>
> Tino
>
>
> 2009/4/3 Maros TIMKO 
>
>> Hi all,
>>
>> we are running CentOS 5.2 Xen virtualization system with the latest CentOS
>> packages with couple of VMs on DELL PowerEdge. "Sometimes" the whole machine
>> freezes without anything in log files, anything on the console. "Sometimes"
>> really means we cannot define why or when. Sometimes the machine was idle
>> with just one VM, sometimes quite busy with couple of VMs.
>>
>>  Has anybody had the same experience? If yes, any hints on how to resolve
>> it or how to trace the cause?
>>
>>
>>
>> Thanks.
>>
>> ___
>> CentOS-virt mailing list
>> CentOS-virt@centos.org
>> http://lists.centos.org/mailman/listinfo/centos-virt
>>
>>
>
> ___
> CentOS-virt mailing list
> CentOS-virt@centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
>
[mccar...@isengard ~]$ ssh xen1
   ___
 __ _ ___ <  /
 \ \ / -_) _ \/ / 
/_\_\\__/_//_/_/  
  
Last login: Mon Apr 27 11:26:22 2009 from isengard.cslabs.clarkson.edu
[mccar...@xen1 ~]$ sudo xm list

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

Password: 
Sorry, try again.
Password: 
Name  ID Mem(MiB) VCPUs State   Time(s)
Domain-0   0 9899 8 r-  12458.2
atp   11  255 1 -b  72748.8
auth   1  127 1 -b121.7
autoguilt  2  255 1 -b523.2
dukr   3  255 1 -b770.4
list   5  255 1 -b191.4
management 6  255 1 -b517.4
osp1   7  255 1 -b 70.4
osp2   8  255 1 -b 68.8
tremulous  9  255 1 -b 397287.4
[mccar...@xen1 ~]$ xm console atp
ERROR Internal error: Could not obtain handle on privileged command interface 
(13 = Permission denied)
Error: Most commands need root access. Please try again as root.
[mccar...@xen1 ~]$ sudo xm console atp
Out of Memory: Kill process 2626 (TreeLimitedRun) score 96585 and children.
Out of memory: Killed process 2627 (spectrum).
Out of Memory: Kill process 2694 (TreeLimitedRun) score 96565 and children.
Out of memory: Killed process 2695 (spectrum).
Out of Memory: Kill process 2914 (TreeLimitedRun) score 96210 and children.
Out of memory: Killed process 2915 (spectrum).
Out of Memory: Kill process 3014 (TreeLimitedRun) score 96153 and children.
Out of memory: Killed process 3015 (spectrum).
Out of Memory: Kill process 3018 (TreeLimitedRun) score 96177 and children.
Out of memory: Killed process 3019 (spectrum).
Out of Memory: Kill process 4466 (spectrum) score 189626 and children.
Out of memory: Killed process 4466 (spectrum).
Out of Memory: Kill process 6324 (TreeLimitedRun) score 96129 and children.
Out of memory: Killed process 6325 (spectrum).
Out of Memory: Kill process 10680 (TreeLimitedRun) score 96