Re: [one-users] Making scheduler allocation aware
The lines
if(scpu=="")
{
scpu="0";
}
were accidentally introduced by me in the code. please ignore them.
On 12 November 2010 12:56, Shashank Rachamalla <
shashank.rachama...@hexagrid.com> wrote:
> Hi
>
> As Ruben pointed out, every time a VM is dispatched, host share counter (
> mem_usage ) gets incremented. Thus there is no way for opennebula to allow
> over committing. However, the problem lies in the following piece of code
> from VirtualMachine.cc where cpu, memory and disk are set to 0 when a VM
> template does not contain "CPU" attribute ( this is true in our case ). The
> mem_usage value in host share does not get incremented because of this. I
> guess, specifying "CPU=0" in the template should solve the problem. I will
> reconfirm after testing.
>
> void VirtualMachine::get_requirements (int& cpu, int& memory, int& disk)
> {
> string scpu;
> istringstream iss;
> float fcpu;
>
> get_template_attribute("MEMORY",memory);
> get_template_attribute("CPU",scpu);
>
> *if ((memory == 0) || (scpu==""))
> {
> cpu= 0;
> memory = 0;
> disk = 0;
>
> return;
> }
> *
> if(scpu=="")
> {
> scpu="0";
> }
>
> iss.str(scpu);
> iss >> fcpu;
>
> cpu= (int) (fcpu * 100);//now in 100%
> memory = memory * 1024; //now in bytes
> disk = 0;
>
> return;
>
> }
>
> On 11 November 2010 19:36, Javier Fontan wrote:
>
>> Hello,
>>
>> Are you sure that those are the exact values for the host? OpenNebula
>> counts "real" (from probes) and allocated (from database) memory so
>> that should not happen.
>>
>> Snippet from a onehost show:
>>
>> --8<--
>> USED MEM (REAL) : 0
>> USED MEM (ALLOCATED) : 65536
>> -->8--
>>
>> I am now working on the kvm monitoring and I have noticed another
>> mismatch even with your probe changes. The values stored in the
>> database for total memory should be changed and that's what I am
>> working on.
>>
>> I am connected to irc.freenode.org in channel #opennebula if you want
>> to discuss this further.
>>
>> Bye
>>
>> On Thu, Nov 11, 2010 at 5:20 AM, Shashank Rachamalla
>> wrote:
>> > Hi Javier
>> >
>> > Thanks for the inputs but I came across another problem while testing:
>> >
>> > If opennebula receives multiple vm requests in a short span of time, the
>> > scheduler might take decisions for all these vms considering the host
>> > monitoring information available from the last monitoring cycle.
>> Ideally,
>> > before processing every pending request, fresh host monitoring
>> information
>> > has to be taken into account as the previous set of requests might have
>> > already changed the host’s state. This can result in over committing
>> when
>> > host is being used close to its full capacity.
>> >
>> > Is there any workaround which helps the scheduler to overcome the above
>> > problem ?
>> >
>> > steps to reproduce the problem scenario:
>> >
>> > Host 1 : Total memory = 3GB
>> > Host 2 : Total memory = 2GB
>> > Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
>> a
>> > higher RANK value )
>> >
>> > VM1: memory = 2GB
>> > VM2: memroy = 2GB
>> >
>> > Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
>> > come up on Host1. ( Thus over committing )
>> >
>> > Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
>> on
>> > Host1 and VM2 will come up on Host2. This is true because opennebula
>> would
>> > have fetched a fresh set of host monitoring information in that time.
>> >
>> >
>> > On 4 November 2010 02:04, Javier Fontan wrote:
>> >>
>> >> Hello,
>> >>
>> >> It looks fine to me. I think that taking out the memory the hypervisor
>> >> may be consuming is key to make it work.
>> >>
>> >> Bye
>> >>
>> >> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>> >> wrote:
>> >> > Javier
>> >> >
>> >> > Yes we are using KVM and OpenNebula 1.4.
>> >> >
>> >> > We have been having this problem since a long time and we were doing
>> all
>> >> > kinds of validations ourselves before submitting the request to
>> >> > OpenNebula.
>> >> > (there should be enough memory in the cloud that matches the
>> requested
>> >> > memory & there should be atleast one host that has memory > requested
>> >> > memory
>> >> > ) We had to do those because OpenNebula would schedule to an
>> arbitrary
>> >> > host based on the existing logic it had.
>> >> > So at last we thought that we need to make OpenNebula aware of memory
>> >> > allocated of running VM's on the host and started this discussion.
>> >> >
>> >> > Thanks for taking up this issue as priority. Appreciate it.
>> >> >
>> >> > Shashank came up with this patch to kvm.rb. Please take a look and
>> let
>> >> > us
>> >> > know if that will work until we get a permanent solution.
>> >> >
>> >> >
>> >> >
>>
>> >> >
>> >> > $mem_allocate
Re: [one-users] Making scheduler allocation aware
Hi
As Ruben pointed out, every time a VM is dispatched, host share counter (
mem_usage ) gets incremented. Thus there is no way for opennebula to allow
over committing. However, the problem lies in the following piece of code
from VirtualMachine.cc where cpu, memory and disk are set to 0 when a VM
template does not contain "CPU" attribute ( this is true in our case ). The
mem_usage value in host share does not get incremented because of this. I
guess, specifying "CPU=0" in the template should solve the problem. I will
reconfirm after testing.
void VirtualMachine::get_requirements (int& cpu, int& memory, int& disk)
{
string scpu;
istringstream iss;
float fcpu;
get_template_attribute("MEMORY",memory);
get_template_attribute("CPU",scpu);
*if ((memory == 0) || (scpu==""))
{
cpu= 0;
memory = 0;
disk = 0;
return;
}
*
if(scpu=="")
{
scpu="0";
}
iss.str(scpu);
iss >> fcpu;
cpu= (int) (fcpu * 100);//now in 100%
memory = memory * 1024; //now in bytes
disk = 0;
return;
}
On 11 November 2010 19:36, Javier Fontan wrote:
> Hello,
>
> Are you sure that those are the exact values for the host? OpenNebula
> counts "real" (from probes) and allocated (from database) memory so
> that should not happen.
>
> Snippet from a onehost show:
>
> --8<--
> USED MEM (REAL) : 0
> USED MEM (ALLOCATED) : 65536
> -->8--
>
> I am now working on the kvm monitoring and I have noticed another
> mismatch even with your probe changes. The values stored in the
> database for total memory should be changed and that's what I am
> working on.
>
> I am connected to irc.freenode.org in channel #opennebula if you want
> to discuss this further.
>
> Bye
>
> On Thu, Nov 11, 2010 at 5:20 AM, Shashank Rachamalla
> wrote:
> > Hi Javier
> >
> > Thanks for the inputs but I came across another problem while testing:
> >
> > If opennebula receives multiple vm requests in a short span of time, the
> > scheduler might take decisions for all these vms considering the host
> > monitoring information available from the last monitoring cycle. Ideally,
> > before processing every pending request, fresh host monitoring
> information
> > has to be taken into account as the previous set of requests might have
> > already changed the host’s state. This can result in over committing when
> > host is being used close to its full capacity.
> >
> > Is there any workaround which helps the scheduler to overcome the above
> > problem ?
> >
> > steps to reproduce the problem scenario:
> >
> > Host 1 : Total memory = 3GB
> > Host 2 : Total memory = 2GB
> > Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
> > higher RANK value )
> >
> > VM1: memory = 2GB
> > VM2: memroy = 2GB
> >
> > Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
> > come up on Host1. ( Thus over committing )
> >
> > Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
> on
> > Host1 and VM2 will come up on Host2. This is true because opennebula
> would
> > have fetched a fresh set of host monitoring information in that time.
> >
> >
> > On 4 November 2010 02:04, Javier Fontan wrote:
> >>
> >> Hello,
> >>
> >> It looks fine to me. I think that taking out the memory the hypervisor
> >> may be consuming is key to make it work.
> >>
> >> Bye
> >>
> >> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
> >> wrote:
> >> > Javier
> >> >
> >> > Yes we are using KVM and OpenNebula 1.4.
> >> >
> >> > We have been having this problem since a long time and we were doing
> all
> >> > kinds of validations ourselves before submitting the request to
> >> > OpenNebula.
> >> > (there should be enough memory in the cloud that matches the
> requested
> >> > memory & there should be atleast one host that has memory > requested
> >> > memory
> >> > ) We had to do those because OpenNebula would schedule to an
> arbitrary
> >> > host based on the existing logic it had.
> >> > So at last we thought that we need to make OpenNebula aware of memory
> >> > allocated of running VM's on the host and started this discussion.
> >> >
> >> > Thanks for taking up this issue as priority. Appreciate it.
> >> >
> >> > Shashank came up with this patch to kvm.rb. Please take a look and let
> >> > us
> >> > know if that will work until we get a permanent solution.
> >> >
> >> >
> >> >
>
> >> >
> >> > $mem_allocated_for_running_vms=0
> >> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
> >> > $dominfo=`virsh dominfo #{i}`
> >> > $dominfo.split(/\n/).each{|line|
> >> > if line.match('^Max memory')
> >> > $mem_allocated_for_running_vms += line.split("
> >> > ")[2].strip.to_i
> >> > end
> >> > }
> >> > end
> >> >
> >> > $mem_used_by_base_hypervisor = [some xy
Re: [one-users] Examples for using Ruby API
If you are using OpenNebula 2.0 you can use the oca gem in your ruby project. https://rubygems.org/gems/oca The API is documented to a certain degree, but missing examples. You can always look at the oca source in the git repo (src/oca/ruby/test/). John On Thu, Nov 11, 2010 at 6:31 PM, KONQUER LABS wrote: > I realize that , how can i use the ruby api from remote systems. > > > On Fri, Nov 12, 2010 at 3:02 AM, Łukasz Oleś wrote: > >> On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: >> > Hi, >> > >> > I am new to opennebula and trying to understand how to use ruby api. I >> > know little scripting. I am looking for some basic examples for >> > opennebula setup info like onehost list, onevm list etc using ruby api. >> >> All of them are already written in ruby api;) >> >> Regards, >> Łukasz Oleś >> > > > ___ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Examples for using Ruby API
I realize that , how can i use the ruby api from remote systems. On Fri, Nov 12, 2010 at 3:02 AM, Łukasz Oleś wrote: > On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: > > Hi, > > > > I am new to opennebula and trying to understand how to use ruby api. I > > know little scripting. I am looking for some basic examples for > > opennebula setup info like onehost list, onevm list etc using ruby api. > > All of them are already written in ruby api;) > > Regards, > Łukasz Oleś > ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Examples for using Ruby API
On Thursday 11 November 2010 20:42:00 KONGQUERLABS wrote: > Hi, > > I am new to opennebula and trying to understand how to use ruby api. I > know little scripting. I am looking for some basic examples for > opennebula setup info like onehost list, onevm list etc using ruby api. All of them are already written in ruby api;) Regards, Łukasz Oleś ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] Examples for using Ruby API
Hi, I am new to opennebula and trying to understand how to use ruby api. I know little scripting. I am looking for some basic examples for opennebula setup info like onehost list, onevm list etc using ruby api. My goal is to build basic web frontend to automate certain cli task using ruby api. Regards ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] [one] opennebula with VMware server 2.0
All, I am trying to Installing opennebula with following combinations Opennebula 2.0 (Ubuntu 10.04) - NFS - VMware server 2.0 I have followed the VMware Driver Guide and installed Libvirt 0.8.3, VMware Server and OpenNebula http://opennebula.org/documentation:rel2.0:evmwareg Now when I am trying to configure the opennebula for VMware server, its not working, I could not able to figure it out where is the issue. So I have couple of question here 1. How to check the complete connectivity between opennebula - libvirt - VMware server for its correctness. 2. Cofiguration documents are not clear e.g in VMware server vmwarerc file should be gsx:// rather esx:// which is never indicated anywhere. So is there any document which talks about all these in finer details. Any kind of help will be appreciated. Thanks -tej ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
[one-users] sparse files as partitions in OpenNebula?
I have a user who has a need for 250GB of disk storage (eventually) that he would like to migrate around with his VM. NFS isn't suitable for this application. This is an application which will start with a file base and then gradually grow. On Amazon this could be a use case for EBS but ONE doesn't have anything like that as far as I can tell. My question, can I create an opennebula template that calls out device "vdc" as a sparse file system eventually growable to 250 GB, and migrate that and save that as necessary? If so, how? We are running opennebula 2.0 and using KVM as our hypervisor. Steve Timm -- -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Computing Division, Scientific Computing Facilities, Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader. ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Fail-over from dead HV node
On 2010. November 11. 15:25:33 Tino Vazquez wrote: > We are working on an extension of the hooks mechanism, so when a host > gets into the ERROR state a script can be triggered with information > about the (allegedly) running VMs in that host, so it can resume them > elsewhere. How do you plan to distinguish VMs that crashed due to host hardware failure from the ones those were shut down by their administrator from inside the WM? Thanks, -- cc > -- > Constantino Vázquez Blanco | dsa-research.org/tinova > Virtualization Technology Engineer / Researcher > OpenNebula Toolkit | opennebula.org > > On Fri, Oct 29, 2010 at 12:03 PM, Oscar Elfving wrote: > > Hi, > > When a hypervisor node dies (or becomes inaccessible), how are you > > supposed to handle the machines that were running on it? ONE of course > > still lists the machines as running since it can't check the machine > > state. How are you guys handling it? In my case I have some single > > purpose webservers running on the nodes and if a hypervisor node dies, I > > would like to have some monitoring software start them up on one of the > > remaining nodes. Best regards, > > Oscar Elfving > > ___ > > Users mailing list > > Users@lists.opennebula.org > > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > ___ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] VMs stuck in PEND state
Hello, Yes, i'm indeed not setting it. Were you able to make the ttylinux image working with Xen? If yes, do you mind sending me a sample configuration? As this is my first time using opennebula, i'm still not aware of how to create VMs properly. Unfortunately, i cannot find any guide or sample tutorials about getting the VM running. Regards, Fernando. Em 11/11/2010, às 06:56, Jaime Melis escreveu: > Hi Fernando, > > you are probably missing the KERNEL and INITRD attributes in your VM template. > Take a look at this: > http://opennebula.org/documentation:rel2.0:template#os_and_boot_options_section > > Regards, > Jaime > > On Wed, Nov 10, 2010 at 5:39 PM, Fernando Morgenstern > wrote: > Hello, > > After applying this patch and adding host again i had a couple of different > errors ( like ruby not being installed in the node ) which i was able to fix. > But now i got stuck again during the VM startup, at the log i see the > following error: > > Wed Nov 10 17:22:38 2010 [LCM][I]: New VM state is BOOT > Wed Nov 10 17:22:38 2010 [VMM][I]: Generating deployment file: > /var/lib/one/5/deployment.0 > Wed Nov 10 17:22:38 2010 [VMM][E]: No kernel or bootloader defined and no > default provided. > Wed Nov 10 17:22:38 2010 [VMM][E]: deploy_action, error generating deployment > file: /var/lib/one/5/deployment.0 > Wed Nov 10 17:22:38 2010 [DiM][I]: New VM state is FAILED > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Deleting > /var/lib/one//5/images > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Executed "rm > -rf /var/lib/one//5/images". > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: TRANSFER SUCCESS 5 - > > The thread that i found here at list about this doesn't contain a solution > for this problem, so i'm not sure how to proceed. > > Regards, > > Fernando. > > Em 10/11/2010, às 11:49, openneb...@nerling.ch escreveu: > > > Yes, it is related to the bug http://dev.opennebula.org/issues/385. > > I attached a patch for /usr/lib/one/mads/one_im_ssh.rb. > > ## Save it on the /tmp > > #cd /usr/lib/one/mads/ > > #patch -p0 < /tmp/one_im_ssh.rb.patch > > > > Quoting Fernando Morgenstern : > > > >> Hi, > >> > >> The tmp folder in my host is empty. > >> > >> Here is the output of the commands: > >> > >> $ onehost show 1 > >> HOST 1 INFORMATION > >> ID: 1 > >> NAME : node01 > >> CLUSTER : default > >> STATE : ERROR > >> IM_MAD: im_xen > >> VM_MAD: vmm_xen > >> TM_MAD: tm_nfs > >> > >> HOST SHARES > >> MAX MEM : 0 > >> USED MEM (REAL) : 0 > >> USED MEM (ALLOCATED) : 0 > >> MAX CPU : 0 > >> USED CPU (REAL) : 0 > >> USED CPU (ALLOCATED) : 0 > >> RUNNING VMS : 0 > >> > >> MONITORING INFORMATION > >> > >> $ onehost show -x 1 > >> > >> 1 > >> node01 > >> 3 > >> im_xen > >> vmm_xen > >> tm_nfs > >> 1289394482 > >> default > >> > >>1 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >> > >> > >> > >> > >> Thanks again for your answers. > >> > >> > >> Em 10/11/2010, às 11:22, openneb...@nerling.ch escreveu: > >> > >>> Hallo Fernando. > >>> try to log in the host and look if there is a folder names /tmp/one. > >>> If not it could be related to the bug: > >>> http://dev.opennebula.org/issues/385 > >>> > >>> please post the output from: > >>> #onehost show 1 > >>> #onehost show -x 1 > >>> > >>> I thought before your host has an id of 0. > >>> > >>> Marlon Nerling > >>> > >>> Quoting Fernando Morgenstern : > >>> > Hello, > > Thanks for the answer. > > You are right, the host is showing an error state and i didn't verified > it. How can i know what is causing the error in host? > > $ onehost list > ID NAME CLUSTER RVM TCPU FCPU ACPUTMEMFMEM > STAT > 1 node01default0 0 0100 0K 0K > err > > $ onevm show 0 > VIRTUAL MACHINE 0 INFORMATION > ID : 0 > NAME : ttylinux > STATE : DONE > LCM_STATE : LCM_INIT > START TIME : 11/09 19:06:37 > END TIME : 11/09 19:11:09 > DEPLOY ID: : - > > VIRTUAL MACHINE MONITORING > NET_RX : 0 > USED MEMORY: 0 > USED CPU : 0 > NET_TX : 0 > > VIRTUAL MACHINE TEMPLATE > CPU=0.1 > DISK=[ > DISK_ID=0, > READONLY=no, > SOURCE=/home/oneadmin/ttylinux.img, > TARGET=hda ] > FEATURES=[ > ACPI=no ] > MEMORY=64 > NAME=ttylinux > NIC=[ > BRIDGE=br0, > IP=*, > MAC=02:00:5d:9f:d0:68, > NETWORK=Small network, > NETWORK_ID=0 ] > VMID=0 > > $ onehost show 0 > Error: [HostInfo] Error getting HOST [0]. >
Re: [one-users] Fail-over from dead HV node
Dear Oscar, We are working on an extension of the hooks mechanism, so when a host gets into the ERROR state a script can be triggered with information about the (allegedly) running VMs in that host, so it can resume them elsewhere. Best regards, -Tino -- Constantino Vázquez Blanco | dsa-research.org/tinova Virtualization Technology Engineer / Researcher OpenNebula Toolkit | opennebula.org On Fri, Oct 29, 2010 at 12:03 PM, Oscar Elfving wrote: > Hi, > When a hypervisor node dies (or becomes inaccessible), how are you supposed > to handle the machines that were running on it? ONE of course still lists > the machines as running since it can't check the machine state. How are you > guys handling it? In my case I have some single purpose webservers running > on the nodes and if a hypervisor node dies, I would like to have some > monitoring software start them up on one of the remaining nodes. > Best regards, > Oscar Elfving > ___ > Users mailing list > Users@lists.opennebula.org > http://lists.opennebula.org/listinfo.cgi/users-opennebula.org > > ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Making scheduler allocation aware
Hello,
Are you sure that those are the exact values for the host? OpenNebula
counts "real" (from probes) and allocated (from database) memory so
that should not happen.
Snippet from a onehost show:
--8<--
USED MEM (REAL) : 0
USED MEM (ALLOCATED) : 65536
-->8--
I am now working on the kvm monitoring and I have noticed another
mismatch even with your probe changes. The values stored in the
database for total memory should be changed and that's what I am
working on.
I am connected to irc.freenode.org in channel #opennebula if you want
to discuss this further.
Bye
On Thu, Nov 11, 2010 at 5:20 AM, Shashank Rachamalla
wrote:
> Hi Javier
>
> Thanks for the inputs but I came across another problem while testing:
>
> If opennebula receives multiple vm requests in a short span of time, the
> scheduler might take decisions for all these vms considering the host
> monitoring information available from the last monitoring cycle. Ideally,
> before processing every pending request, fresh host monitoring information
> has to be taken into account as the previous set of requests might have
> already changed the host’s state. This can result in over committing when
> host is being used close to its full capacity.
>
> Is there any workaround which helps the scheduler to overcome the above
> problem ?
>
> steps to reproduce the problem scenario:
>
> Host 1 : Total memory = 3GB
> Host 2 : Total memory = 2GB
> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
> higher RANK value )
>
> VM1: memory = 2GB
> VM2: memroy = 2GB
>
> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
> come up on Host1. ( Thus over committing )
>
> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on
> Host1 and VM2 will come up on Host2. This is true because opennebula would
> have fetched a fresh set of host monitoring information in that time.
>
>
> On 4 November 2010 02:04, Javier Fontan wrote:
>>
>> Hello,
>>
>> It looks fine to me. I think that taking out the memory the hypervisor
>> may be consuming is key to make it work.
>>
>> Bye
>>
>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>> wrote:
>> > Javier
>> >
>> > Yes we are using KVM and OpenNebula 1.4.
>> >
>> > We have been having this problem since a long time and we were doing all
>> > kinds of validations ourselves before submitting the request to
>> > OpenNebula.
>> > (there should be enough memory in the cloud that matches the requested
>> > memory & there should be atleast one host that has memory > requested
>> > memory
>> > ) We had to do those because OpenNebula would schedule to an arbitrary
>> > host based on the existing logic it had.
>> > So at last we thought that we need to make OpenNebula aware of memory
>> > allocated of running VM's on the host and started this discussion.
>> >
>> > Thanks for taking up this issue as priority. Appreciate it.
>> >
>> > Shashank came up with this patch to kvm.rb. Please take a look and let
>> > us
>> > know if that will work until we get a permanent solution.
>> >
>> >
>> >
>> >
>> > $mem_allocated_for_running_vms=0
>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
>> > $dominfo=`virsh dominfo #{i}`
>> > $dominfo.split(/\n/).each{|line|
>> > if line.match('^Max memory')
>> > $mem_allocated_for_running_vms += line.split("
>> > ")[2].strip.to_i
>> > end
>> > }
>> > end
>> >
>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set aside
>> > for
>> > hypervisor]
>> >
>> > $free_memory = $total_memory.to_i - (
>> > $mem_allocated_for_running_vms.to_i +
>> > $mem_used_by_base_hypervisor.to_i )
>> >
>> >
>> > ==
>> >
>> > Ranga
>> >
>> > On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan wrote:
>> >>
>> >> Hello,
>> >>
>> >> Sorry for the delay in the response.
>> >>
>> >> It looks that the problem is OpenNebula calculating available memory.
>> >> For xen >= 3.2 there is a reliable way to get available memory that is
>> >> calling "xm info" and getting "max_free_memory" attribute.
>> >> Unfortunately for kvm or xen < 3.2 there is not such attribute. I
>> >> suppose you are using kvm as you tell about "free" command.
>> >>
>> >> I began analyzing the kvm IM probe that gets memory information and
>> >> there is a problem on the way to get total memory. Here is how it now
>> >> gets memory information:
>> >>
>> >> TOTALMEMORY: runs virsh info that gets the real physical memory
>> >> installed in the machine
>> >> FREEMEMORY: runs free command and gets the free column data without
>> >> buffers and cache
>> >> USEDMEMORY: runs top command and gets used memory from it (this counts
>> >> buffers and cache)
>> >>
>> >> This is a big problem as those values do not match one with another (I
>> >> don't really
Re: [one-users] Making scheduler allocation aware
Hi Yes that is true for 1.4, OpenNebula 2.0 also updates the running vms each time a VM is scheduled... Cheers Ruben On Thu, Nov 11, 2010 at 12:58 PM, Shashank Rachamalla wrote: > Hi > > What Marlon said regarding RANK=RUNNING_VMS is the exact concern. However, I > will check out the source code once. We are using opennebula 1.4 with the > following template: ( Note that the value of PRIORITY is calculated using > FREEMEMORY, FREECPU and RUNNING_VMS ) > > DISK=[ > clone=no, > source=/mnt/dev_store_10/images/glpnnu0, > target=hda, > type=disk > ] > DISK=[ > clone=no, > > source=/mnt/dev_store_10/iso/FABCAC11-768B-4683-EB99-085ECB80, > target=hdb, > type=cdrom > ] > MEMORY=2000 > REQUIREMENTS="FREEMEMORY>2048000" > RANK=PRIORITY > NAME=glpnnu > OS=[ > boot=cdrom > ] > NIC=[ > model=e1000, > bridge=br101, > MAC=00:89:10:36:10:26 > ] > INPUT=[ > type=tablet > ] > VCPU=1 > GRAPHICS=[ > port=5971, > type=vnc, > listen=0.0.0.0 > ] > > On 11 November 2010 16:08, wrote: >> >> Hi Rubens. >> I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario, because >> the scheduler does not update the RUNNING_VMS after the creation of the VM >> but just after monitoring the host! >> So between this events the RUNNING_VMS value stays unchanged and by my >> experience this Host will become the 'chosen one' for new deployed VMs up to >> the next host monitoring. >> And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED) and >> the memory used by the VM to prevent overcommiting, we could look in the >> source code for this. >> I must say never have experienced a host who became more VMs as possible. >> >> Best regards >> >> Marlon >> >> Zitat von "Ruben S. Montero" : >> >>> Hi, >>> >>> We use the total memory to allocate VM's, this is not going to change >>> between monitoring events. Right now, we are able to schedule multiple >>> VMs in the same scheduling step, to the same host without >>> overcommitting memory. >>> >>> Cheers >>> >>> On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla >>> wrote: Hi Thanks for the reply. I agree that a VM will not be allowed to use more than what has been specified in its template during creation but I was referring to a scenario where memory available on the host is over committed. I guess the problem here is with allowing multiple VMs to be dispatched on to a single host in one schedule cycle. On 11 November 2010 14:47, Ruben S. Montero wrote: > > Hi, > > Regarding capacity (CPU, Memory), it is updated every time a VM is > submitted, so no overcommitment is possible (beyond that specified by > the CPU attribute in the VM template). This also works in 1.4 > > Cheers > > Ruben > > On Thu, Nov 11, 2010 at 10:08 AM, wrote: > > Hallo Shashank. > > I'm having the same problem in 1.4. > > You must workaround it by youself, so instead of using onevm > > directly, > > use > > it with a wrapper script who looks and waits for deploy if a VM is > > pending. > > I hope this behaviour is fixed on 2.0 (Hallo developers??) > > > > Best regards > > > > Marlon > > Zitat von Shashank Rachamalla : > > > >> Hi Javier > >> > >> Thanks for the inputs but I came across another problem while > >> testing: > >> > >> If opennebula receives multiple vm requests in a short span of time, > >> the > >> scheduler might take decisions for all these vms considering the > >> host > >> monitoring information available from the last monitoring cycle. > >> Ideally, > >> before processing every pending request, fresh host monitoring > >> information > >> has to be taken into account as the previous set of requests might > >> have > >> already changed the host’s state. This can result in over committing > >> when > >> host is being used close to its full capacity. > >> > >> *Is there any workaround which helps the scheduler to overcome the > >> above > >> problem ?* > >> > >> steps to reproduce the problem scenario: > >> > >> Host 1 : Total memory = 3GB > >> Host 2 : Total memory = 2GB > >> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will > >> have > >> a > >> higher RANK value ) > >> > >> VM1: memory = 2GB > >> VM2: memroy = 2GB > >> > >> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 > >> will > >> come up on Host1. ( Thus over committing ) > >> > >> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come > >> up > >> on > >> Host1 and VM2 will come up on Host2. This is true because opennebula > >> would > >> have fetched a fresh set of host monitoring inform
Re: [one-users] Making scheduler allocation aware
Hi What Marlon said regarding RANK=RUNNING_VMS is the exact concern. However, I will check out the source code once. We are using opennebula 1.4 with the following template: ( Note that the value of PRIORITY is calculated using FREEMEMORY, FREECPU and RUNNING_VMS ) DISK=[ clone=no, source=/mnt/dev_store_10/images/glpnnu0, target=hda, type=disk ] DISK=[ clone=no, source=/mnt/dev_store_10/iso/FABCAC11-768B-4683-EB99-085ECB80, target=hdb, type=cdrom ] MEMORY=2000 REQUIREMENTS="FREEMEMORY>2048000" RANK=PRIORITY NAME=glpnnu OS=[ boot=cdrom ] NIC=[ model=e1000, bridge=br101, MAC=00:89:10:36:10:26 ] INPUT=[ type=tablet ] VCPU=1 GRAPHICS=[ port=5971, type=vnc, listen=0.0.0.0 ] On 11 November 2010 16:08, wrote: > Hi Rubens. > I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario, because > the scheduler does not update the RUNNING_VMS after the creation of the VM > but just after monitoring the host! > So between this events the RUNNING_VMS value stays unchanged and by my > experience this Host will become the 'chosen one' for new deployed VMs up to > the next host monitoring. > And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED) and > the memory used by the VM to prevent overcommiting, we could look in the > source code for this. > I must say never have experienced a host who became more VMs as possible. > > Best regards > > Marlon > > Zitat von "Ruben S. Montero" : > > > Hi, >> >> We use the total memory to allocate VM's, this is not going to change >> between monitoring events. Right now, we are able to schedule multiple >> VMs in the same scheduling step, to the same host without >> overcommitting memory. >> >> Cheers >> >> On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla >> wrote: >> >>> Hi >>> >>> Thanks for the reply. I agree that a VM will not be allowed to use more >>> than >>> what has been specified in its template during creation but I was >>> referring >>> to a scenario where memory available on the host is over committed. I >>> guess >>> the problem here is with allowing multiple VMs to be dispatched on to a >>> single host in one schedule cycle. >>> >>> >>> On 11 November 2010 14:47, Ruben S. Montero >>> wrote: >>> Hi, Regarding capacity (CPU, Memory), it is updated every time a VM is submitted, so no overcommitment is possible (beyond that specified by the CPU attribute in the VM template). This also works in 1.4 Cheers Ruben On Thu, Nov 11, 2010 at 10:08 AM, wrote: > Hallo Shashank. > I'm having the same problem in 1.4. > You must workaround it by youself, so instead of using onevm directly, > use > it with a wrapper script who looks and waits for deploy if a VM is > pending. > I hope this behaviour is fixed on 2.0 (Hallo developers??) > > Best regards > > Marlon > Zitat von Shashank Rachamalla : > >> Hi Javier >> >> Thanks for the inputs but I came across another problem while testing: >> >> If opennebula receives multiple vm requests in a short span of time, >> the >> scheduler might take decisions for all these vms considering the host >> monitoring information available from the last monitoring cycle. >> Ideally, >> before processing every pending request, fresh host monitoring >> information >> has to be taken into account as the previous set of requests might have >> already changed the host’s state. This can result in over committing >> when >> host is being used close to its full capacity. >> >> *Is there any workaround which helps the scheduler to overcome the >> above >> problem ?* >> >> steps to reproduce the problem scenario: >> >> Host 1 : Total memory = 3GB >> Host 2 : Total memory = 2GB >> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have >> a >> higher RANK value ) >> >> VM1: memory = 2GB >> VM2: memroy = 2GB >> >> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 >> will >> come up on Host1. ( Thus over committing ) >> >> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up >> on >> Host1 and VM2 will come up on Host2. This is true because opennebula >> would >> have fetched a fresh set of host monitoring information in that time. >> >> >> On 4 November 2010 02:04, Javier Fontan wrote: >> >>> Hello, >>> >>> It looks fine to me. I think that taking out the memory the hypervisor >>> may be consuming is key to make it work. >>> >>> Bye >>> >>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula >>> wrote: >>> > Javier >>> > >
Re: [one-users] Making scheduler allocation aware
Hi Rubens.
I'm sure the RANK=-RUNNING_VMS will not apply in such a scenario,
because the scheduler does not update the RUNNING_VMS after the
creation of the VM but just after monitoring the host!
So between this events the RUNNING_VMS value stays unchanged and by my
experience this Host will become the 'chosen one' for new deployed VMs
up to the next host monitoring.
And i'm not really sure if the scheduler sums USED MEMORY (ALLOCATED)
and the memory used by the VM to prevent overcommiting, we could look
in the source code for this.
I must say never have experienced a host who became more VMs as possible.
Best regards
Marlon
Zitat von "Ruben S. Montero" :
Hi,
We use the total memory to allocate VM's, this is not going to change
between monitoring events. Right now, we are able to schedule multiple
VMs in the same scheduling step, to the same host without
overcommitting memory.
Cheers
On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla
wrote:
Hi
Thanks for the reply. I agree that a VM will not be allowed to use more than
what has been specified in its template during creation but I was referring
to a scenario where memory available on the host is over committed. I guess
the problem here is with allowing multiple VMs to be dispatched on to a
single host in one schedule cycle.
On 11 November 2010 14:47, Ruben S. Montero wrote:
Hi,
Regarding capacity (CPU, Memory), it is updated every time a VM is
submitted, so no overcommitment is possible (beyond that specified by
the CPU attribute in the VM template). This also works in 1.4
Cheers
Ruben
On Thu, Nov 11, 2010 at 10:08 AM, wrote:
> Hallo Shashank.
> I'm having the same problem in 1.4.
> You must workaround it by youself, so instead of using onevm directly,
> use
> it with a wrapper script who looks and waits for deploy if a VM is
> pending.
> I hope this behaviour is fixed on 2.0 (Hallo developers??)
>
> Best regards
>
> Marlon
> Zitat von Shashank Rachamalla :
>
>> Hi Javier
>>
>> Thanks for the inputs but I came across another problem while testing:
>>
>> If opennebula receives multiple vm requests in a short span of time,
>> the
>> scheduler might take decisions for all these vms considering the host
>> monitoring information available from the last monitoring cycle.
>> Ideally,
>> before processing every pending request, fresh host monitoring
>> information
>> has to be taken into account as the previous set of requests might have
>> already changed the host’s state. This can result in over committing
>> when
>> host is being used close to its full capacity.
>>
>> *Is there any workaround which helps the scheduler to overcome the
>> above
>> problem ?*
>>
>> steps to reproduce the problem scenario:
>>
>> Host 1 : Total memory = 3GB
>> Host 2 : Total memory = 2GB
>> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
>> a
>> higher RANK value )
>>
>> VM1: memory = 2GB
>> VM2: memroy = 2GB
>>
>> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2
>> will
>> come up on Host1. ( Thus over committing )
>>
>> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
>> on
>> Host1 and VM2 will come up on Host2. This is true because opennebula
>> would
>> have fetched a fresh set of host monitoring information in that time.
>>
>>
>> On 4 November 2010 02:04, Javier Fontan wrote:
>>
>>> Hello,
>>>
>>> It looks fine to me. I think that taking out the memory the hypervisor
>>> may be consuming is key to make it work.
>>>
>>> Bye
>>>
>>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>>> wrote:
>>> > Javier
>>> >
>>> > Yes we are using KVM and OpenNebula 1.4.
>>> >
>>> > We have been having this problem since a long time and we were doing
>>> > all
>>> > kinds of validations ourselves before submitting the request to
>>> OpenNebula.
>>> > (there should be enough memory in the cloud that matches the
>>> > requested
>>> > memory & there should be atleast one host that has memory >
>>> > requested
>>> memory
>>> > ) We had to do those because OpenNebula would schedule to an
>>> > arbitrary
>>> > host based on the existing logic it had.
>>> > So at last we thought that we need to make OpenNebula aware of
>>> > memory
>>> > allocated of running VM's on the host and started this discussion.
>>> >
>>> > Thanks for taking up this issue as priority. Appreciate it.
>>> >
>>> > Shashank came up with this patch to kvm.rb. Please take a look and
>>> > let
>>> > us
>>> > know if that will work until we get a permanent solution.
>>> >
>>> >
>>>
>>>
>>>
>>> >
>>> > $mem_allocated_for_running_vms=0
>>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
>>> > $dominfo=`virsh dominfo #{i}`
>>> > $dominfo.split(/\n/).each{|line|
>>> > if line.match('^Max memory')
>>> > $mem_allocated_for_running_vms += line.split("
>>> > ")[2].strip.to_i
>
Re: [one-users] Making scheduler allocation aware
Hi,
We use the total memory to allocate VM's, this is not going to change
between monitoring events. Right now, we are able to schedule multiple
VMs in the same scheduling step, to the same host without
overcommitting memory.
Cheers
On Thu, Nov 11, 2010 at 10:42 AM, Shashank Rachamalla
wrote:
> Hi
>
> Thanks for the reply. I agree that a VM will not be allowed to use more than
> what has been specified in its template during creation but I was referring
> to a scenario where memory available on the host is over committed. I guess
> the problem here is with allowing multiple VMs to be dispatched on to a
> single host in one schedule cycle.
>
>
> On 11 November 2010 14:47, Ruben S. Montero wrote:
>>
>> Hi,
>>
>> Regarding capacity (CPU, Memory), it is updated every time a VM is
>> submitted, so no overcommitment is possible (beyond that specified by
>> the CPU attribute in the VM template). This also works in 1.4
>>
>> Cheers
>>
>> Ruben
>>
>> On Thu, Nov 11, 2010 at 10:08 AM, wrote:
>> > Hallo Shashank.
>> > I'm having the same problem in 1.4.
>> > You must workaround it by youself, so instead of using onevm directly,
>> > use
>> > it with a wrapper script who looks and waits for deploy if a VM is
>> > pending.
>> > I hope this behaviour is fixed on 2.0 (Hallo developers??)
>> >
>> > Best regards
>> >
>> > Marlon
>> > Zitat von Shashank Rachamalla :
>> >
>> >> Hi Javier
>> >>
>> >> Thanks for the inputs but I came across another problem while testing:
>> >>
>> >> If opennebula receives multiple vm requests in a short span of time,
>> >> the
>> >> scheduler might take decisions for all these vms considering the host
>> >> monitoring information available from the last monitoring cycle.
>> >> Ideally,
>> >> before processing every pending request, fresh host monitoring
>> >> information
>> >> has to be taken into account as the previous set of requests might have
>> >> already changed the host’s state. This can result in over committing
>> >> when
>> >> host is being used close to its full capacity.
>> >>
>> >> *Is there any workaround which helps the scheduler to overcome the
>> >> above
>> >> problem ?*
>> >>
>> >> steps to reproduce the problem scenario:
>> >>
>> >> Host 1 : Total memory = 3GB
>> >> Host 2 : Total memory = 2GB
>> >> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
>> >> a
>> >> higher RANK value )
>> >>
>> >> VM1: memory = 2GB
>> >> VM2: memroy = 2GB
>> >>
>> >> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2
>> >> will
>> >> come up on Host1. ( Thus over committing )
>> >>
>> >> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
>> >> on
>> >> Host1 and VM2 will come up on Host2. This is true because opennebula
>> >> would
>> >> have fetched a fresh set of host monitoring information in that time.
>> >>
>> >>
>> >> On 4 November 2010 02:04, Javier Fontan wrote:
>> >>
>> >>> Hello,
>> >>>
>> >>> It looks fine to me. I think that taking out the memory the hypervisor
>> >>> may be consuming is key to make it work.
>> >>>
>> >>> Bye
>> >>>
>> >>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>> >>> wrote:
>> >>> > Javier
>> >>> >
>> >>> > Yes we are using KVM and OpenNebula 1.4.
>> >>> >
>> >>> > We have been having this problem since a long time and we were doing
>> >>> > all
>> >>> > kinds of validations ourselves before submitting the request to
>> >>> OpenNebula.
>> >>> > (there should be enough memory in the cloud that matches the
>> >>> > requested
>> >>> > memory & there should be atleast one host that has memory >
>> >>> > requested
>> >>> memory
>> >>> > ) We had to do those because OpenNebula would schedule to an
>> >>> > arbitrary
>> >>> > host based on the existing logic it had.
>> >>> > So at last we thought that we need to make OpenNebula aware of
>> >>> > memory
>> >>> > allocated of running VM's on the host and started this discussion.
>> >>> >
>> >>> > Thanks for taking up this issue as priority. Appreciate it.
>> >>> >
>> >>> > Shashank came up with this patch to kvm.rb. Please take a look and
>> >>> > let
>> >>> > us
>> >>> > know if that will work until we get a permanent solution.
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> >
>> >>> > $mem_allocated_for_running_vms=0
>> >>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
>> >>> > $dominfo=`virsh dominfo #{i}`
>> >>> > $dominfo.split(/\n/).each{|line|
>> >>> > if line.match('^Max memory')
>> >>> > $mem_allocated_for_running_vms += line.split("
>> >>> > ")[2].strip.to_i
>> >>> > end
>> >>> > }
>> >>> > end
>> >>> >
>> >>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set
>> >>> > aside
>> >>> > for
>> >>> > hypervisor]
>> >>> >
>> >>> > $free_memory = $total_memory.to_i - (
>> >>> > $mem_allocated_for_running_vms.to_i
>> >>> +
>> >>> > $mem_used_by_base_hypervisor.to_
Re: [one-users] Making scheduler allocation aware
Hi
Thanks for the reply. I agree that a VM will not be allowed to use more than
what has been specified in its template during creation but I was referring
to a scenario where memory available on the host is over committed. I guess
the problem here is with allowing multiple VMs to be dispatched on to a
single host in one schedule cycle.
On 11 November 2010 14:47, Ruben S. Montero wrote:
> Hi,
>
> Regarding capacity (CPU, Memory), it is updated every time a VM is
> submitted, so no overcommitment is possible (beyond that specified by
> the CPU attribute in the VM template). This also works in 1.4
>
> Cheers
>
> Ruben
>
> On Thu, Nov 11, 2010 at 10:08 AM, wrote:
> > Hallo Shashank.
> > I'm having the same problem in 1.4.
> > You must workaround it by youself, so instead of using onevm directly,
> use
> > it with a wrapper script who looks and waits for deploy if a VM is
> pending.
> > I hope this behaviour is fixed on 2.0 (Hallo developers??)
> >
> > Best regards
> >
> > Marlon
> > Zitat von Shashank Rachamalla :
> >
> >> Hi Javier
> >>
> >> Thanks for the inputs but I came across another problem while testing:
> >>
> >> If opennebula receives multiple vm requests in a short span of time, the
> >> scheduler might take decisions for all these vms considering the host
> >> monitoring information available from the last monitoring cycle.
> Ideally,
> >> before processing every pending request, fresh host monitoring
> >> information
> >> has to be taken into account as the previous set of requests might have
> >> already changed the host’s state. This can result in over committing
> when
> >> host is being used close to its full capacity.
> >>
> >> *Is there any workaround which helps the scheduler to overcome the above
> >> problem ?*
> >>
> >> steps to reproduce the problem scenario:
> >>
> >> Host 1 : Total memory = 3GB
> >> Host 2 : Total memory = 2GB
> >> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have
> a
> >> higher RANK value )
> >>
> >> VM1: memory = 2GB
> >> VM2: memroy = 2GB
> >>
> >> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
> >> come up on Host1. ( Thus over committing )
> >>
> >> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up
> on
> >> Host1 and VM2 will come up on Host2. This is true because opennebula
> would
> >> have fetched a fresh set of host monitoring information in that time.
> >>
> >>
> >> On 4 November 2010 02:04, Javier Fontan wrote:
> >>
> >>> Hello,
> >>>
> >>> It looks fine to me. I think that taking out the memory the hypervisor
> >>> may be consuming is key to make it work.
> >>>
> >>> Bye
> >>>
> >>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
> >>> wrote:
> >>> > Javier
> >>> >
> >>> > Yes we are using KVM and OpenNebula 1.4.
> >>> >
> >>> > We have been having this problem since a long time and we were doing
> >>> > all
> >>> > kinds of validations ourselves before submitting the request to
> >>> OpenNebula.
> >>> > (there should be enough memory in the cloud that matches the
> requested
> >>> > memory & there should be atleast one host that has memory > requested
> >>> memory
> >>> > ) We had to do those because OpenNebula would schedule to an
> >>> > arbitrary
> >>> > host based on the existing logic it had.
> >>> > So at last we thought that we need to make OpenNebula aware of memory
> >>> > allocated of running VM's on the host and started this discussion.
> >>> >
> >>> > Thanks for taking up this issue as priority. Appreciate it.
> >>> >
> >>> > Shashank came up with this patch to kvm.rb. Please take a look and
> let
> >>> > us
> >>> > know if that will work until we get a permanent solution.
> >>> >
> >>> >
> >>>
> >>>
>
> >>> >
> >>> > $mem_allocated_for_running_vms=0
> >>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
> >>> > $dominfo=`virsh dominfo #{i}`
> >>> > $dominfo.split(/\n/).each{|line|
> >>> > if line.match('^Max memory')
> >>> > $mem_allocated_for_running_vms += line.split("
> >>> > ")[2].strip.to_i
> >>> > end
> >>> > }
> >>> > end
> >>> >
> >>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set aside
> >>> > for
> >>> > hypervisor]
> >>> >
> >>> > $free_memory = $total_memory.to_i - (
> >>> > $mem_allocated_for_running_vms.to_i
> >>> +
> >>> > $mem_used_by_base_hypervisor.to_i )
> >>> >
> >>> >
> >>>
> >>>
> ==
> >>> >
> >>> > Ranga
> >>> >
> >>> > On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan
> >>> > wrote:
> >>> >>
> >>> >> Hello,
> >>> >>
> >>> >> Sorry for the delay in the response.
> >>> >>
> >>> >> It looks that the problem is OpenNebula calculating available
> memory.
> >>> >> For xen >= 3.2 there is a reliable way to get available memory that
> is
> >>> >> calling "xm info" and getting "max_fre
Re: [one-users] Making scheduler allocation aware
Hi,
Regarding capacity (CPU, Memory), it is updated every time a VM is
submitted, so no overcommitment is possible (beyond that specified by
the CPU attribute in the VM template). This also works in 1.4
Cheers
Ruben
On Thu, Nov 11, 2010 at 10:08 AM, wrote:
> Hallo Shashank.
> I'm having the same problem in 1.4.
> You must workaround it by youself, so instead of using onevm directly, use
> it with a wrapper script who looks and waits for deploy if a VM is pending.
> I hope this behaviour is fixed on 2.0 (Hallo developers??)
>
> Best regards
>
> Marlon
> Zitat von Shashank Rachamalla :
>
>> Hi Javier
>>
>> Thanks for the inputs but I came across another problem while testing:
>>
>> If opennebula receives multiple vm requests in a short span of time, the
>> scheduler might take decisions for all these vms considering the host
>> monitoring information available from the last monitoring cycle. Ideally,
>> before processing every pending request, fresh host monitoring
>> information
>> has to be taken into account as the previous set of requests might have
>> already changed the host’s state. This can result in over committing when
>> host is being used close to its full capacity.
>>
>> *Is there any workaround which helps the scheduler to overcome the above
>> problem ?*
>>
>> steps to reproduce the problem scenario:
>>
>> Host 1 : Total memory = 3GB
>> Host 2 : Total memory = 2GB
>> Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
>> higher RANK value )
>>
>> VM1: memory = 2GB
>> VM2: memroy = 2GB
>>
>> Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
>> come up on Host1. ( Thus over committing )
>>
>> Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on
>> Host1 and VM2 will come up on Host2. This is true because opennebula would
>> have fetched a fresh set of host monitoring information in that time.
>>
>>
>> On 4 November 2010 02:04, Javier Fontan wrote:
>>
>>> Hello,
>>>
>>> It looks fine to me. I think that taking out the memory the hypervisor
>>> may be consuming is key to make it work.
>>>
>>> Bye
>>>
>>> On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
>>> wrote:
>>> > Javier
>>> >
>>> > Yes we are using KVM and OpenNebula 1.4.
>>> >
>>> > We have been having this problem since a long time and we were doing
>>> > all
>>> > kinds of validations ourselves before submitting the request to
>>> OpenNebula.
>>> > (there should be enough memory in the cloud that matches the requested
>>> > memory & there should be atleast one host that has memory > requested
>>> memory
>>> > ) We had to do those because OpenNebula would schedule to an
>>> > arbitrary
>>> > host based on the existing logic it had.
>>> > So at last we thought that we need to make OpenNebula aware of memory
>>> > allocated of running VM's on the host and started this discussion.
>>> >
>>> > Thanks for taking up this issue as priority. Appreciate it.
>>> >
>>> > Shashank came up with this patch to kvm.rb. Please take a look and let
>>> > us
>>> > know if that will work until we get a permanent solution.
>>> >
>>> >
>>>
>>>
>>> >
>>> > $mem_allocated_for_running_vms=0
>>> > for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
>>> > $dominfo=`virsh dominfo #{i}`
>>> > $dominfo.split(/\n/).each{|line|
>>> > if line.match('^Max memory')
>>> > $mem_allocated_for_running_vms += line.split("
>>> > ")[2].strip.to_i
>>> > end
>>> > }
>>> > end
>>> >
>>> > $mem_used_by_base_hypervisor = [some xyz kb that we want to set aside
>>> > for
>>> > hypervisor]
>>> >
>>> > $free_memory = $total_memory.to_i - (
>>> > $mem_allocated_for_running_vms.to_i
>>> +
>>> > $mem_used_by_base_hypervisor.to_i )
>>> >
>>> >
>>>
>>> ==
>>> >
>>> > Ranga
>>> >
>>> > On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan
>>> > wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> Sorry for the delay in the response.
>>> >>
>>> >> It looks that the problem is OpenNebula calculating available memory.
>>> >> For xen >= 3.2 there is a reliable way to get available memory that is
>>> >> calling "xm info" and getting "max_free_memory" attribute.
>>> >> Unfortunately for kvm or xen < 3.2 there is not such attribute. I
>>> >> suppose you are using kvm as you tell about "free" command.
>>> >>
>>> >> I began analyzing the kvm IM probe that gets memory information and
>>> >> there is a problem on the way to get total memory. Here is how it now
>>> >> gets memory information:
>>> >>
>>> >> TOTALMEMORY: runs virsh info that gets the real physical memory
>>> >> installed in the machine
>>> >> FREEMEMORY: runs free command and gets the free column data without
>>> >> buffers and cache
>>> >> USEDMEMORY: runs top command and gets used memory from it (this counts
>>> >> buffers and cache)
>>> >>
>>>
Re: [one-users] Making scheduler allocation aware
Hallo Shashank.
I'm having the same problem in 1.4.
You must workaround it by youself, so instead of using onevm directly,
use it with a wrapper script who looks and waits for deploy if a VM is
pending.
I hope this behaviour is fixed on 2.0 (Hallo developers??)
Best regards
Marlon
Zitat von Shashank Rachamalla :
Hi Javier
Thanks for the inputs but I came across another problem while testing:
If opennebula receives multiple vm requests in a short span of time, the
scheduler might take decisions for all these vms considering the host
monitoring information available from the last monitoring cycle. Ideally,
before processing every pending request, fresh host monitoring information
has to be taken into account as the previous set of requests might have
already changed the host’s state. This can result in over committing when
host is being used close to its full capacity.
*Is there any workaround which helps the scheduler to overcome the above
problem ?*
steps to reproduce the problem scenario:
Host 1 : Total memory = 3GB
Host 2 : Total memory = 2GB
Assume Host1 and Host2 have same number of CPU cores. ( Host1 will have a
higher RANK value )
VM1: memory = 2GB
VM2: memroy = 2GB
Start VM1 and VM2 immediately one after the other. Both VM1 and VM2 will
come up on Host1. ( Thus over committing )
Start VM1 and VM2 with an intermediate delay of 60sec. VM1 will come up on
Host1 and VM2 will come up on Host2. This is true because opennebula would
have fetched a fresh set of host monitoring information in that time.
On 4 November 2010 02:04, Javier Fontan wrote:
Hello,
It looks fine to me. I think that taking out the memory the hypervisor
may be consuming is key to make it work.
Bye
On Wed, Nov 3, 2010 at 8:32 PM, Rangababu Chakravarthula
wrote:
> Javier
>
> Yes we are using KVM and OpenNebula 1.4.
>
> We have been having this problem since a long time and we were doing all
> kinds of validations ourselves before submitting the request to
OpenNebula.
> (there should be enough memory in the cloud that matches the requested
> memory & there should be atleast one host that has memory > requested
memory
> ) We had to do those because OpenNebula would schedule to an arbitrary
> host based on the existing logic it had.
> So at last we thought that we need to make OpenNebula aware of memory
> allocated of running VM's on the host and started this discussion.
>
> Thanks for taking up this issue as priority. Appreciate it.
>
> Shashank came up with this patch to kvm.rb. Please take a look and let us
> know if that will work until we get a permanent solution.
>
>
>
> $mem_allocated_for_running_vms=0
> for i in `virsh list|grep running|tr -s ' ' ' '|cut -f2 -d' '` do
> $dominfo=`virsh dominfo #{i}`
> $dominfo.split(/\n/).each{|line|
> if line.match('^Max memory')
> $mem_allocated_for_running_vms += line.split("
> ")[2].strip.to_i
> end
> }
> end
>
> $mem_used_by_base_hypervisor = [some xyz kb that we want to set aside for
> hypervisor]
>
> $free_memory = $total_memory.to_i - ( $mem_allocated_for_running_vms.to_i
+
> $mem_used_by_base_hypervisor.to_i )
>
>
==
>
> Ranga
>
> On Wed, Nov 3, 2010 at 2:16 PM, Javier Fontan wrote:
>>
>> Hello,
>>
>> Sorry for the delay in the response.
>>
>> It looks that the problem is OpenNebula calculating available memory.
>> For xen >= 3.2 there is a reliable way to get available memory that is
>> calling "xm info" and getting "max_free_memory" attribute.
>> Unfortunately for kvm or xen < 3.2 there is not such attribute. I
>> suppose you are using kvm as you tell about "free" command.
>>
>> I began analyzing the kvm IM probe that gets memory information and
>> there is a problem on the way to get total memory. Here is how it now
>> gets memory information:
>>
>> TOTALMEMORY: runs virsh info that gets the real physical memory
>> installed in the machine
>> FREEMEMORY: runs free command and gets the free column data without
>> buffers and cache
>> USEDMEMORY: runs top command and gets used memory from it (this counts
>> buffers and cache)
>>
>> This is a big problem as those values do not match one with another (I
>> don't really know how I failed to see this before). Here is the
>> monitoring data from a host without VMs.
>>
>> --8<--
>> TOTALMEMORY=8193988
>> USEDMEMORY=7819952
>> FREEMEMORY=7911924
>> -->8--
>>
>> As you can see it makes no sense at all. Even the TOTALMEMORY that is
>> got from virsh info is very misleading for oned as the host linux
>> instance does not have access to all that memory (some is consumed by
>> the hypervisor itself) as seen calling a free command:
>>
>> --8<--
>> total used free sharedbuffers cached
>> Mem: 81939887819192 374796 0 64176
7473992
>>
Re: [one-users] VMs stuck in PEND state
Hi Fernando, you are probably missing the KERNEL and INITRD attributes in your VM template. Take a look at this: http://opennebula.org/documentation:rel2.0:template#os_and_boot_options_section Regards, Jaime On Wed, Nov 10, 2010 at 5:39 PM, Fernando Morgenstern < ferna...@consultorpc.com> wrote: > Hello, > > After applying this patch and adding host again i had a couple of different > errors ( like ruby not being installed in the node ) which i was able to > fix. But now i got stuck again during the VM startup, at the log i see the > following error: > > Wed Nov 10 17:22:38 2010 [LCM][I]: New VM state is BOOT > Wed Nov 10 17:22:38 2010 [VMM][I]: Generating deployment file: > /var/lib/one/5/deployment.0 > Wed Nov 10 17:22:38 2010 [VMM][E]: No kernel or bootloader defined and no > default provided. > Wed Nov 10 17:22:38 2010 [VMM][E]: deploy_action, error generating > deployment file: /var/lib/one/5/deployment.0 > Wed Nov 10 17:22:38 2010 [DiM][I]: New VM state is FAILED > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Deleting > /var/lib/one//5/images > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: LOG - 5 tm_delete.sh: Executed > "rm -rf /var/lib/one//5/images". > Wed Nov 10 17:22:38 2010 [TM][W]: Ignored: TRANSFER SUCCESS 5 - > > The thread that i found here at list about this doesn't contain a solution > for this problem, so i'm not sure how to proceed. > > Regards, > > Fernando. > > Em 10/11/2010, às 11:49, openneb...@nerling.ch escreveu: > > > Yes, it is related to the bug http://dev.opennebula.org/issues/385. > > I attached a patch for /usr/lib/one/mads/one_im_ssh.rb. > > ## Save it on the /tmp > > #cd /usr/lib/one/mads/ > > #patch -p0 < /tmp/one_im_ssh.rb.patch > > > > Quoting Fernando Morgenstern : > > > >> Hi, > >> > >> The tmp folder in my host is empty. > >> > >> Here is the output of the commands: > >> > >> $ onehost show 1 > >> HOST 1 INFORMATION > >> ID: 1 > >> NAME : node01 > >> CLUSTER : default > >> STATE : ERROR > >> IM_MAD: im_xen > >> VM_MAD: vmm_xen > >> TM_MAD: tm_nfs > >> > >> HOST SHARES > >> MAX MEM : 0 > >> USED MEM (REAL) : 0 > >> USED MEM (ALLOCATED) : 0 > >> MAX CPU : 0 > >> USED CPU (REAL) : 0 > >> USED CPU (ALLOCATED) : 0 > >> RUNNING VMS : 0 > >> > >> MONITORING INFORMATION > >> > >> $ onehost show -x 1 > >> > >> 1 > >> node01 > >> 3 > >> im_xen > >> vmm_xen > >> tm_nfs > >> 1289394482 > >> default > >> > >>1 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >>0 > >> > >> > >> > >> > >> Thanks again for your answers. > >> > >> > >> Em 10/11/2010, às 11:22, openneb...@nerling.ch escreveu: > >> > >>> Hallo Fernando. > >>> try to log in the host and look if there is a folder names /tmp/one. > >>> If not it could be related to the bug: > http://dev.opennebula.org/issues/385 > >>> > >>> please post the output from: > >>> #onehost show 1 > >>> #onehost show -x 1 > >>> > >>> I thought before your host has an id of 0. > >>> > >>> Marlon Nerling > >>> > >>> Quoting Fernando Morgenstern : > >>> > Hello, > > Thanks for the answer. > > You are right, the host is showing an error state and i didn't > verified it. How can i know what is causing the error in host? > > $ onehost list > ID NAME CLUSTER RVM TCPU FCPU ACPUTMEMFMEM > STAT > 1 node01default0 0 0100 0K 0K > err > > $ onevm show 0 > VIRTUAL MACHINE 0 INFORMATION > ID : 0 > NAME : ttylinux > STATE : DONE > LCM_STATE : LCM_INIT > START TIME : 11/09 19:06:37 > END TIME : 11/09 19:11:09 > DEPLOY ID: : - > > VIRTUAL MACHINE MONITORING > NET_RX : 0 > USED MEMORY: 0 > USED CPU : 0 > NET_TX : 0 > > VIRTUAL MACHINE TEMPLATE > CPU=0.1 > DISK=[ > DISK_ID=0, > READONLY=no, > SOURCE=/home/oneadmin/ttylinux.img, > TARGET=hda ] > FEATURES=[ > ACPI=no ] > MEMORY=64 > NAME=ttylinux > NIC=[ > BRIDGE=br0, > IP=*, > MAC=02:00:5d:9f:d0:68, > NETWORK=Small network, > NETWORK_ID=0 ] > VMID=0 > > $ onehost show 0 > Error: [HostInfo] Error getting HOST [0]. > > Thanks! > > Em 10/11/2010, às 06:45, openneb...@nerling.ch escreveu: > > > Hallo Fernando. > > Could you please post the output of: > > #onehost list > > #onevm show 0 > > #onehost show 0 > > > > It seems that none of your Hosts are enabled! > > Tue Nov 9 20:31:18 2010 [HOST][D]: Discovered Hosts (enabled): > > > > best regards > > > > Marlon Nerling > > > > Zitat
