Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-03 Thread Gilad Chaplik 
> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Omer Frenkel" , "Eli Mesika" ,
> "engine-devel" 
> Sent: Thursday, April 3, 2014 5:27:56 PM
> Subject: Re: [Engine-devel] vds_dynamic refactor

> True but that's no reason to have a bad schema
I'm open to new ideas and truly want to understand what is bad? 

> On Apr 3, 2014 5:18 PM, "Gilad Chaplik" < gchap...@redhat.com > wrote:

> > - Original Message -
> 
> > > From: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > To: "Gilad Chaplik" < gchap...@redhat.com >
> 
> > > Cc: "Eli Mesika" < emes...@redhat.com >, "engine-devel" <
> > > engine-de...@ovirt.org >
> 
> > > Sent: Thursday, April 3, 2014 5:16:51 PM
> 
> > > Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> > >
> 
> > > Don't go down that road. Status shouldn't be saved in the db.
> 
> > > But anyway statistics is rapidly changing. So it fits...
> 

> > First let's have a notion of caching, then notion of shared caching, then I
> > can start thinking of not going down that road...
> 

> > > On Apr 3, 2014 5:14 PM, "Gilad Chaplik" < gchap...@redhat.com > wrote:
> 
> > >
> 
> > > > - Original Message -
> 
> > > > > From: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > > > To: "Eli Mesika" < emes...@redhat.com >
> 
> > > > > Cc: "Gilad Chaplik" < gchap...@redhat.com >, "engine-devel" <
> 
> > > > engine-de...@ovirt.org >
> 
> > > > > Sent: Thursday, April 3, 2014 4:36:07 PM
> 
> > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> > > > >
> 
> > > > > I would be happy if we can lose vds_dynamic all together, and just
> > > > > keep
> 
> > > > > vds_static and vds_statistics. Our performance will be happy too ;-)
> 
> > > > >
> 
> > > >
> 
> > > > @Liran, status and pending fields are very fragile ones, IMO need
> > > > separate
> 
> > > > table.
> 
> > > > @Eli, iirc, you don't have a problem with adding more tables :-)
> 
> > > >
> 
> > > > >
> 
> > > > > On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika < emes...@redhat.com >
> > > > > wrote:
> 
> > > > >
> 
> > > > > >
> 
> > > > > >
> 
> > > > > > - Original Message -
> 
> > > > > > > From: "Gilad Chaplik" < gchap...@redhat.com >
> 
> > > > > > > To: "Yair Zaslavsky" < yzasl...@redhat.com >
> 
> > > > > > > Cc: "engine-devel" < engine-de...@ovirt.org >
> 
> > > > > > > Sent: Thursday, April 3, 2014 4:00:25 PM
> 
> > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> > > > > > >
> 
> > > > > > > - Original Message -
> 
> > > > > > > > From: "Yair Zaslavsky" < yzasl...@redhat.com >
> 
> > > > > > > > To: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > > > > > > Cc: "Gilad Chaplik" < gchap...@redhat.com >, "engine-devel"
> 
> > > > > > > > < engine-de...@ovirt.org >
> 
> > > > > > > > Sent: Thursday, April 3, 2014 2:12:59 PM
> 
> > > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> > > > > > > >
> 
> > > > > > > >
> 
> > > > > > > >
> 
> > > > > > > > - Original Message -
> 
> > > > > > > > > From: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > > > > > > > To: "Gilad Chaplik" < gchap...@redhat.com >
> 
> > > > > > > > > Cc: "engine-devel" < engine-de...@ovirt.org >
> 
> > > > > > > > > Sent: Thursday, April 3, 2014 2:04:29 PM
> 
> > > > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> > > > > > > > >
> 
> > > > > > > > > Why not move it to vds_static?
> 
> > > > > > > >
> 
> > > > > > > > +1 on Liran's comment.
> 
> > > > > >
> 
> > > > > > +1 as well , vds_static is the place for that
> 
> > > > > >
> 
> > > > > > > > I would prefer not to add more complexity to the vds tables
> 
> > > > family.
> 
> > > > > > > > Such complexity may effect performs of queries/views.
> 
> > > > > > > > If you wish, you can create a view on top of vds_static named
> 
> > > > > > vds_on_boot
> 
> > > > > > > > for
> 
> > > > > > > > querying of vds on boot info.
> 
> > > > > > > >
> 
> > > > > > > > Yair
> 
> > > > > > >
> 
> > > > > > > That means moving almost all of vds_dynamic into vds_static
> > > > > > > except
> > > > > > > of
> 
> > > > > > memory,
> 
> > > > > > > pending resources and status (maybe more but not much);
> 
> > > > > > > and there will not be any db separation between user input and
> 
> > > > on_boot
> 
> > > > > > data.
> 
> > > > > >
> 
> > > > > > Why we should have such separation ?
> 
> > > > > >
> 
> > > > > > >
> 
> > > > > > > Thanks,
> 
> > > > > > > Gilad.
> 
> > > > > > >
> 
> > > > > > > >
> 
> > > > > > > > >
> 
> > > > > > > > >
> 
> > > > > > > > > On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik <
> 
> > > > gchap...@redhat.com >
> 
> > > > > > > > > wrote:
> 
> > > > > > > > >
> 
> > > > > > > > > > Hi list,
> 
> > > > > > > > > >
> 
> > > > > > > > > > I propose to split vds_dynamic table into 2 tables:
> 
> > > > > > > > > > - vds_dynamic
> 
> > > > > > > > > > - vds_on_boot
> 
> > > > > > > > > > We need a place to put all non-dynamic data that gets
> > > > > > > > > > updated
> 
> > > > once
> 
> > > > > > the
> 
> > > > > >

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-03 Thread Gilad Chaplik 
- Original Message -
> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Omer Frenkel" , "Eli Mesika" , 
> "engine-devel" ,
> de...@linode01.ovirt.org
> Sent: Thursday, April 3, 2014 7:51:39 PM
> Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik  wrote:
> 
> > *From: *"Liran Zelkha" 
> > *To: *"Gilad Chaplik" 
> > *Cc: *"Omer Frenkel" , "Eli Mesika" <
> > emes...@redhat.com>, "engine-devel" 
> > *Sent: *Thursday, April 3, 2014 5:27:56 PM
> > *Subject: *Re: [Engine-devel] vds_dynamic refactor
> >
> > True but that's no reason to have a bad schema
> >
> > I'm open to new ideas and truly want to understand what is bad?
> >
> The problem is with both updates and selects.
> For selects - to get all the information for the VDS we have multiple
> joins. Adding another one will hurt performance even more.

What about creating the VDS list in the db. i.e. your patch [1] about 
constructing VDS objects in the engine, should occur in the db within a SP, and 
should be transparent to the server. that will solve the multiple table join.

[1] http://gerrit.ovirt.org/#/c/21662/

> For updates - we have vds_static thats hardly changed. vds_statistics that
> changes all the time. vds_dynamic is not changed allot - but is updated all
> the time because of the status. I think it's best to split it to the two
> existing tables (BTW - relevant for VM as well)
> 

- As omer stated in a separate thread static stores users config.
- Updating status/pending resources in statistics sounds very racy.

saying that, I think I have a valid argument for vds_on_boot table.

Thanks, 
Gilad.


> > On Apr 3, 2014 5:18 PM, "Gilad Chaplik"  wrote:
> >
> >> - Original Message -
> >> > From: "Liran Zelkha" 
> >> > To: "Gilad Chaplik" 
> >> > Cc: "Eli Mesika" , "engine-devel" <
> >> engine-de...@ovirt.org>
> >> > Sent: Thursday, April 3, 2014 5:16:51 PM
> >> > Subject: Re: [Engine-devel] vds_dynamic refactor
> >> >
> >> > Don't go down that road.  Status shouldn't be saved in the db.
> >> > But anyway statistics is rapidly changing. So it fits...
> >>
> >> First let's have a notion of caching, then notion of shared caching, then
> >> I can start thinking of not going down that road...
> >>
> >> > On Apr 3, 2014 5:14 PM, "Gilad Chaplik"  wrote:
> >> >
> >> > > - Original Message -
> >> > > > From: "Liran Zelkha" 
> >> > > > To: "Eli Mesika" 
> >> > > > Cc: "Gilad Chaplik" , "engine-devel" <
> >> > > engine-de...@ovirt.org>
> >> > > > Sent: Thursday, April 3, 2014 4:36:07 PM
> >> > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> >> > > >
> >> > > > I would be happy if we can lose vds_dynamic all together, and just
> >> keep
> >> > > > vds_static and vds_statistics. Our performance will be happy too ;-)
> >> > > >
> >> > >
> >> > > @Liran, status and pending fields are very fragile ones, IMO need
> >> separate
> >> > > table.
> >> > > @Eli, iirc, you don't have a problem with adding more tables :-)
> >> > >
> >> > > >
> >> > > > On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika 
> >> wrote:
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > - Original Message -
> >> > > > > > From: "Gilad Chaplik" 
> >> > > > > > To: "Yair Zaslavsky" 
> >> > > > > > Cc: "engine-devel" 
> >> > > > > > Sent: Thursday, April 3, 2014 4:00:25 PM
> >> > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> >> > > > > >
> >> > > > > > - Original Message -
> >> > > > > > > From: "Yair Zaslavsky" 
> >> > > > > > > To: "Liran Zelkha" 
> >> > > > > > > Cc: "Gilad Chaplik" , "engine-devel"
> >> > > > > > > 
> >> > > > > > > Sent: Thursday, April 3, 2014 2:12:59 PM
> >> > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > - Original Message -
> >> > > > > > > > From: "Liran Zelkha" 
> >> > > > > > > > To: "Gilad Chaplik" 
> >> > > > > > > > Cc: "engine-devel" 
> >> > > > > > > > Sent: Thursday, April 3, 2014 2:04:29 PM
> >> > > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> >> > > > > > > >
> >> > > > > > > > Why not move it to vds_static?
> >> > > > > > >
> >> > > > > > > +1 on Liran's comment.
> >> > > > >
> >> > > > > +1 as well , vds_static is the place for that
> >> > > > >
> >> > > > > > > I would prefer not to add more complexity to the  vds tables
> >> > > family.
> >> > > > > > > Such complexity may effect performs of queries/views.
> >> > > > > > > If you wish, you can create a view on top of vds_static named
> >> > > > > vds_on_boot
> >> > > > > > > for
> >> > > > > > > querying of vds on boot info.
> >> > > > > > >
> >> > > > > > > Yair
> >> > > > > >
> >> > > > > > That means moving almost all of vds_dynamic into vds_static
> >> except of
> >> > > > > memory,
> >> > > > > > pending resources and status (maybe more but not much);
> >> > > > > > and there will not be any db separation between user input and
> >> > > on_boot
> >> > > > > data.
> >> > > > >
> >> > > > > Why we s

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-04 Thread Liran Zelkha 
On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik  wrote:

> *From: *"Liran Zelkha" 
> *To: *"Gilad Chaplik" 
> *Cc: *"Omer Frenkel" , "Eli Mesika" <
> emes...@redhat.com>, "engine-devel" 
> *Sent: *Thursday, April 3, 2014 5:27:56 PM
> *Subject: *Re: [Engine-devel] vds_dynamic refactor
>
> True but that's no reason to have a bad schema
>
> I'm open to new ideas and truly want to understand what is bad?
>
The problem is with both updates and selects.
For selects - to get all the information for the VDS we have multiple
joins. Adding another one will hurt performance even more.
For updates - we have vds_static thats hardly changed. vds_statistics that
changes all the time. vds_dynamic is not changed allot - but is updated all
the time because of the status. I think it's best to split it to the two
existing tables (BTW - relevant for VM as well)

> On Apr 3, 2014 5:18 PM, "Gilad Chaplik"  wrote:
>
>> - Original Message -
>> > From: "Liran Zelkha" 
>> > To: "Gilad Chaplik" 
>> > Cc: "Eli Mesika" , "engine-devel" <
>> engine-de...@ovirt.org>
>> > Sent: Thursday, April 3, 2014 5:16:51 PM
>> > Subject: Re: [Engine-devel] vds_dynamic refactor
>> >
>> > Don't go down that road.  Status shouldn't be saved in the db.
>> > But anyway statistics is rapidly changing. So it fits...
>>
>> First let's have a notion of caching, then notion of shared caching, then
>> I can start thinking of not going down that road...
>>
>> > On Apr 3, 2014 5:14 PM, "Gilad Chaplik"  wrote:
>> >
>> > > - Original Message -
>> > > > From: "Liran Zelkha" 
>> > > > To: "Eli Mesika" 
>> > > > Cc: "Gilad Chaplik" , "engine-devel" <
>> > > engine-de...@ovirt.org>
>> > > > Sent: Thursday, April 3, 2014 4:36:07 PM
>> > > > Subject: Re: [Engine-devel] vds_dynamic refactor
>> > > >
>> > > > I would be happy if we can lose vds_dynamic all together, and just
>> keep
>> > > > vds_static and vds_statistics. Our performance will be happy too ;-)
>> > > >
>> > >
>> > > @Liran, status and pending fields are very fragile ones, IMO need
>> separate
>> > > table.
>> > > @Eli, iirc, you don't have a problem with adding more tables :-)
>> > >
>> > > >
>> > > > On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika 
>> wrote:
>> > > >
>> > > > >
>> > > > >
>> > > > > - Original Message -
>> > > > > > From: "Gilad Chaplik" 
>> > > > > > To: "Yair Zaslavsky" 
>> > > > > > Cc: "engine-devel" 
>> > > > > > Sent: Thursday, April 3, 2014 4:00:25 PM
>> > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
>> > > > > >
>> > > > > > - Original Message -
>> > > > > > > From: "Yair Zaslavsky" 
>> > > > > > > To: "Liran Zelkha" 
>> > > > > > > Cc: "Gilad Chaplik" , "engine-devel"
>> > > > > > > 
>> > > > > > > Sent: Thursday, April 3, 2014 2:12:59 PM
>> > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > - Original Message -
>> > > > > > > > From: "Liran Zelkha" 
>> > > > > > > > To: "Gilad Chaplik" 
>> > > > > > > > Cc: "engine-devel" 
>> > > > > > > > Sent: Thursday, April 3, 2014 2:04:29 PM
>> > > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
>> > > > > > > >
>> > > > > > > > Why not move it to vds_static?
>> > > > > > >
>> > > > > > > +1 on Liran's comment.
>> > > > >
>> > > > > +1 as well , vds_static is the place for that
>> > > > >
>> > > > > > > I would prefer not to add more complexity to the  vds tables
>> > > family.
>> > > > > > > Such complexity may effect performs of queries/views.
>> > > > > > > If you wish, you can create a view on top of vds_static named
>> > > > > vds_on_boot
>> > > > > > > for
>> > > > > > > querying of vds on boot info.
>> > > > > > >
>> > > > > > > Yair
>> > > > > >
>> > > > > > That means moving almost all of vds_dynamic into vds_static
>> except of
>> > > > > memory,
>> > > > > > pending resources and status (maybe more but not much);
>> > > > > > and there will not be any db separation between user input and
>> > > on_boot
>> > > > > data.
>> > > > >
>> > > > > Why we should have such separation ?
>> > > > >
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Gilad.
>> > > > > >
>> > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik <
>> > > gchap...@redhat.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi list,
>> > > > > > > > >
>> > > > > > > > > I propose to split vds_dynamic table into 2 tables:
>> > > > > > > > > - vds_dynamic
>> > > > > > > > > - vds_on_boot
>> > > > > > > > > We need a place to put all non-dynamic data that gets
>> updated
>> > > once
>> > > > > the
>> > > > > > > > > host is booted, and I think dynamic isn't the place for
>> it.
>> > > > > > > > > In first phase we will put there NUMA host topoplogy, but
>> > > later on
>> > > > > > > > > migrate
>> > > > > > > > > all non-dynamic host data (cpu, os, etc.).
>> > > > > > > > >
>> > > > > > > > > thoughts?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > Gilad.
>> > > >

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-04 Thread Liran Zelkha 
On Thu, Apr 3, 2014 at 8:40 PM, Gilad Chaplik  wrote:

> - Original Message -
> > From: "Liran Zelkha" 
> > To: "Gilad Chaplik" 
> > Cc: "Omer Frenkel" , "Eli Mesika" <
> emes...@redhat.com>, "engine-devel" ,
> > de...@linode01.ovirt.org
> > Sent: Thursday, April 3, 2014 7:51:39 PM
> > Subject: Re: [Engine-devel] vds_dynamic refactor
> >
> > On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik 
> wrote:
> >
> > > *From: *"Liran Zelkha" 
> > > *To: *"Gilad Chaplik" 
> > > *Cc: *"Omer Frenkel" , "Eli Mesika" <
> > > emes...@redhat.com>, "engine-devel" 
> > > *Sent: *Thursday, April 3, 2014 5:27:56 PM
> > > *Subject: *Re: [Engine-devel] vds_dynamic refactor
> > >
> > > True but that's no reason to have a bad schema
> > >
> > > I'm open to new ideas and truly want to understand what is bad?
> > >
> > The problem is with both updates and selects.
> > For selects - to get all the information for the VDS we have multiple
> > joins. Adding another one will hurt performance even more.
>
> What about creating the VDS list in the db. i.e. your patch [1] about
> constructing VDS objects in the engine, should occur in the db within a SP,
> and should be transparent to the server. that will solve the multiple table
> join.
>

The joins are happening on the server. We have a VDS view that brings in
information from many tables and we retrieve rows from it all the time.
Just run an explain on a select * from vds and see for yourself.

>
> [1] http://gerrit.ovirt.org/#/c/21662/
>
> > For updates - we have vds_static thats hardly changed. vds_statistics
> that
> > changes all the time. vds_dynamic is not changed allot - but is updated
> all
> > the time because of the status. I think it's best to split it to the two
> > existing tables (BTW - relevant for VM as well)
> >
>
> - As omer stated in a separate thread static stores users config.
> - Updating status/pending resources in statistics sounds very racy.
>
> saying that, I think I have a valid argument for vds_on_boot table.
>
> Thanks,
> Gilad.
>
>
> > > On Apr 3, 2014 5:18 PM, "Gilad Chaplik"  wrote:
> > >
> > >> - Original Message -
> > >> > From: "Liran Zelkha" 
> > >> > To: "Gilad Chaplik" 
> > >> > Cc: "Eli Mesika" , "engine-devel" <
> > >> engine-de...@ovirt.org>
> > >> > Sent: Thursday, April 3, 2014 5:16:51 PM
> > >> > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >> >
> > >> > Don't go down that road.  Status shouldn't be saved in the db.
> > >> > But anyway statistics is rapidly changing. So it fits...
> > >>
> > >> First let's have a notion of caching, then notion of shared caching,
> then
> > >> I can start thinking of not going down that road...
> > >>
> > >> > On Apr 3, 2014 5:14 PM, "Gilad Chaplik" 
> wrote:
> > >> >
> > >> > > - Original Message -
> > >> > > > From: "Liran Zelkha" 
> > >> > > > To: "Eli Mesika" 
> > >> > > > Cc: "Gilad Chaplik" , "engine-devel" <
> > >> > > engine-de...@ovirt.org>
> > >> > > > Sent: Thursday, April 3, 2014 4:36:07 PM
> > >> > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >> > > >
> > >> > > > I would be happy if we can lose vds_dynamic all together, and
> just
> > >> keep
> > >> > > > vds_static and vds_statistics. Our performance will be happy
> too ;-)
> > >> > > >
> > >> > >
> > >> > > @Liran, status and pending fields are very fragile ones, IMO need
> > >> separate
> > >> > > table.
> > >> > > @Eli, iirc, you don't have a problem with adding more tables :-)
> > >> > >
> > >> > > >
> > >> > > > On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika 
> > >> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > - Original Message -
> > >> > > > > > From: "Gilad Chaplik" 
> > >> > > > > > To: "Yair Zaslavsky" 
> > >> > > > > > Cc: "engine-devel" 
> > >> > > > > > Sent: Thursday, April 3, 2014 4:00:25 PM
> > >> > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >> > > > > >
> > >> > > > > > - Original Message -
> > >> > > > > > > From: "Yair Zaslavsky" 
> > >> > > > > > > To: "Liran Zelkha" 
> > >> > > > > > > Cc: "Gilad Chaplik" , "engine-devel"
> > >> > > > > > > 
> > >> > > > > > > Sent: Thursday, April 3, 2014 2:12:59 PM
> > >> > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > - Original Message -
> > >> > > > > > > > From: "Liran Zelkha" 
> > >> > > > > > > > To: "Gilad Chaplik" 
> > >> > > > > > > > Cc: "engine-devel" 
> > >> > > > > > > > Sent: Thursday, April 3, 2014 2:04:29 PM
> > >> > > > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >> > > > > > > >
> > >> > > > > > > > Why not move it to vds_static?
> > >> > > > > > >
> > >> > > > > > > +1 on Liran's comment.
> > >> > > > >
> > >> > > > > +1 as well , vds_static is the place for that
> > >> > > > >
> > >> > > > > > > I would prefer not to add more complexity to the  vds
> tables
> > >> > > family.
> > >> > > > > > > Such complexity may effect performs of queries/views.
> > >> > > > > > > If

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-04 Thread Itamar Heim 

On 04/04/2014 04:30 AM, Liran Zelkha wrote:




On Thu, Apr 3, 2014 at 8:40 PM, Gilad Chaplik mailto:gchap...@redhat.com>> wrote:

- Original Message -
 > From: "Liran Zelkha" mailto:liran.zel...@gmail.com>>
 > To: "Gilad Chaplik" mailto:gchap...@redhat.com>>
 > Cc: "Omer Frenkel" mailto:ofren...@redhat.com>>, "Eli Mesika" mailto:emes...@redhat.com>>, "engine-devel" mailto:engine-de...@ovirt.org>>,
 > de...@linode01.ovirt.org 
 > Sent: Thursday, April 3, 2014 7:51:39 PM
 > Subject: Re: [Engine-devel] vds_dynamic refactor
 >
 > On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik
mailto:gchap...@redhat.com>> wrote:
 >
 > > *From: *"Liran Zelkha" mailto:liran.zel...@gmail.com>>
 > > *To: *"Gilad Chaplik" mailto:gchap...@redhat.com>>
 > > *Cc: *"Omer Frenkel" mailto:ofren...@redhat.com>>, "Eli Mesika" <
 > > emes...@redhat.com >, "engine-devel"
mailto:engine-de...@ovirt.org>>
 > > *Sent: *Thursday, April 3, 2014 5:27:56 PM
 > > *Subject: *Re: [Engine-devel] vds_dynamic refactor
 > >
 > > True but that's no reason to have a bad schema
 > >
 > > I'm open to new ideas and truly want to understand what is bad?
 > >
 > The problem is with both updates and selects.
 > For selects - to get all the information for the VDS we have multiple
 > joins. Adding another one will hurt performance even more.

What about creating the VDS list in the db. i.e. your patch [1]
about constructing VDS objects in the engine, should occur in the db
within a SP, and should be transparent to the server. that will
solve the multiple table join.


The joins are happening on the server. We have a VDS view that brings in
information from many tables and we retrieve rows from it all the time.
Just run an explain on a select * from vds and see for yourself.


thought the distinction was vds_static is updated by user 
(configuration), where as vds_dyanmic/statistics is reported from vdsm 
(slow/fast updates).


please remember joining them to one table, also means DWH will ETA all 
of data each time. today it will only copy statistics if dynamic did not 
change.


___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-04 Thread Liran Zelkha 
On Fri, Apr 4, 2014 at 3:15 PM, Itamar Heim  wrote:

> On 04/04/2014 04:30 AM, Liran Zelkha wrote:
>
>>
>>
>>
>> On Thu, Apr 3, 2014 at 8:40 PM, Gilad Chaplik > > wrote:
>>
>> - Original Message -
>>  > From: "Liran Zelkha" > >
>>  > To: "Gilad Chaplik" > >
>>  > Cc: "Omer Frenkel" > >, "Eli Mesika" > >, "engine-devel" > >,
>>  > de...@linode01.ovirt.org 
>>  > Sent: Thursday, April 3, 2014 7:51:39 PM
>>  > Subject: Re: [Engine-devel] vds_dynamic refactor
>>  >
>>  > On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik
>> mailto:gchap...@redhat.com>> wrote:
>>  >
>>  > > *From: *"Liran Zelkha" > >
>>
>>  > > *To: *"Gilad Chaplik" > >
>>
>>  > > *Cc: *"Omer Frenkel" > >, "Eli Mesika" <
>>  > > emes...@redhat.com >, "engine-devel"
>> mailto:engine-de...@ovirt.org>>
>>
>>  > > *Sent: *Thursday, April 3, 2014 5:27:56 PM
>>  > > *Subject: *Re: [Engine-devel] vds_dynamic refactor
>>  > >
>>  > > True but that's no reason to have a bad schema
>>  > >
>>  > > I'm open to new ideas and truly want to understand what is bad?
>>  > >
>>  > The problem is with both updates and selects.
>>  > For selects - to get all the information for the VDS we have
>> multiple
>>  > joins. Adding another one will hurt performance even more.
>>
>> What about creating the VDS list in the db. i.e. your patch [1]
>> about constructing VDS objects in the engine, should occur in the db
>> within a SP, and should be transparent to the server. that will
>> solve the multiple table join.
>>
>>
>> The joins are happening on the server. We have a VDS view that brings in
>> information from many tables and we retrieve rows from it all the time.
>> Just run an explain on a select * from vds and see for yourself.
>>
>
> thought the distinction was vds_static is updated by user (configuration),
> where as vds_dyanmic/statistics is reported from vdsm (slow/fast updates).
>
> True - vds_static hardly change. But vds_dynamic is split-brain - it has
practically static data, but also has the status - which changes rapidly.
And since we have 3 tables, we need to do 3 joins to get VDS (actually we
do much more).


> please remember joining them to one table, also means DWH will ETA all of
> data each time. today it will only copy statistics if dynamic did not
> change.
>
> I'm not saying joining all 3 tables to 1 table, just make them 2 tables.
And since the data is hardly changing - same logic of DWH will stay.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Itamar Heim 

On 04/03/2014 07:51 PM, Liran Zelkha wrote:

The problem is with both updates and selects.
For selects - to get all the information for the VDS we have multiple
joins. Adding another one will hurt performance even more.
For updates - we have vds_static thats hardly changed. vds_statistics
that changes all the time. vds_dynamic is not changed allot - but is
updated all the time because of the status. I think it's best to split
it to the two existing tables (BTW - relevant for VM as well)


but we don't update it unless the status has changed, which is a rare 
occurance?

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Liran Zelkha 
On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  wrote:

> On 04/03/2014 07:51 PM, Liran Zelkha wrote:
>
>> The problem is with both updates and selects.
>> For selects - to get all the information for the VDS we have multiple
>> joins. Adding another one will hurt performance even more.
>> For updates - we have vds_static thats hardly changed. vds_statistics
>> that changes all the time. vds_dynamic is not changed allot - but is
>> updated all the time because of the status. I think it's best to split
>> it to the two existing tables (BTW - relevant for VM as well)
>>
>
> but we don't update it unless the status has changed, which is a rare
> occurance?
>
Actually - no. We can definitely see times we are updating vds_dynamic with
no reason at all. I tried to create patches for that - but it happens from
many different places in the code.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Itamar Heim 

On 04/06/2014 11:32 AM, Liran Zelkha wrote:




On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim mailto:ih...@redhat.com>> wrote:

On 04/03/2014 07:51 PM, Liran Zelkha wrote:

The problem is with both updates and selects.
For selects - to get all the information for the VDS we have
multiple
joins. Adding another one will hurt performance even more.
For updates - we have vds_static thats hardly changed.
vds_statistics
that changes all the time. vds_dynamic is not changed allot - but is
updated all the time because of the status. I think it's best to
split
it to the two existing tables (BTW - relevant for VM as well)


but we don't update it unless the status has changed, which is a
rare occurance?

Actually - no. We can definitely see times we are updating vds_dynamic
with no reason at all. I tried to create patches for that - but it
happens from many different places in the code.


what would be updated vds_dyanmic for status not originating in update 
run time info?

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Gilad Chaplik 
- Original Message -
> From: "Itamar Heim" 
> To: "Liran Zelkha" 
> Cc: "Gilad Chaplik" , de...@linode01.ovirt.org, 
> "engine-devel" 
> Sent: Sunday, April 6, 2014 11:33:12 AM
> Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> >
> >
> >
> > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > wrote:
> >
> > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> >
> > The problem is with both updates and selects.
> > For selects - to get all the information for the VDS we have
> > multiple
> > joins. Adding another one will hurt performance even more.
> > For updates - we have vds_static thats hardly changed.
> > vds_statistics
> > that changes all the time. vds_dynamic is not changed allot - but
> > is
> > updated all the time because of the status. I think it's best to
> > split
> > it to the two existing tables (BTW - relevant for VM as well)
> >
> >
> > but we don't update it unless the status has changed, which is a
> > rare occurance?
> >
> > Actually - no. We can definitely see times we are updating vds_dynamic
> > with no reason at all. I tried to create patches for that - but it
> > happens from many different places in the code.
> 
> what would be updated vds_dyanmic for status not originating in update
> run time info?

We have separate DB flows for that (updateStatus and 
updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
A question: do you know if we update status in updateVdsDynamic? :-) not sure 
but I found a possible race for pending resources (cpu, mem), LOL :-)

Still holds my original thought for having vds_on_boot.

> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Liran Zelkha 
On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik  wrote:

> - Original Message -
> > From: "Itamar Heim" 
> > To: "Liran Zelkha" 
> > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> "engine-devel" 
> > Sent: Sunday, April 6, 2014 11:33:12 AM
> > Subject: Re: [Engine-devel] vds_dynamic refactor
> >
> > On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> > >
> > >
> > >
> > > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > > wrote:
> > >
> > > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> > >
> > > The problem is with both updates and selects.
> > > For selects - to get all the information for the VDS we have
> > > multiple
> > > joins. Adding another one will hurt performance even more.
> > > For updates - we have vds_static thats hardly changed.
> > > vds_statistics
> > > that changes all the time. vds_dynamic is not changed allot -
> but
> > > is
> > > updated all the time because of the status. I think it's best
> to
> > > split
> > > it to the two existing tables (BTW - relevant for VM as well)
> > >
> > >
> > > but we don't update it unless the status has changed, which is a
> > > rare occurance?
> > >
> > > Actually - no. We can definitely see times we are updating vds_dynamic
> > > with no reason at all. I tried to create patches for that - but it
> > > happens from many different places in the code.
> >
> > what would be updated vds_dyanmic for status not originating in update
> > run time info?
>
> We have separate DB flows for that (updateStatus and
> updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
> A question: do you know if we update status in updateVdsDynamic? :-) not
> sure but I found a possible race for pending resources (cpu, mem), LOL :-)
>
> I think we do but not sure. Will check.


> Still holds my original thought for having vds_on_boot.
>
>
>
Let's talk f2f on Tuesday?
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Gilad Chaplik 
- Original Message -
> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Itamar Heim" , de...@linode01.ovirt.org, 
> "engine-devel" 
> Sent: Sunday, April 6, 2014 3:26:24 PM
> Subject: Re: [Engine-devel] vds_dynamic refactor
> 
> On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik  wrote:
> 
> > - Original Message -
> > > From: "Itamar Heim" 
> > > To: "Liran Zelkha" 
> > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> > "engine-devel" 
> > > Sent: Sunday, April 6, 2014 11:33:12 AM
> > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >
> > > On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> > > >
> > > >
> > > >
> > > > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > > > wrote:
> > > >
> > > > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> > > >
> > > > The problem is with both updates and selects.
> > > > For selects - to get all the information for the VDS we have
> > > > multiple
> > > > joins. Adding another one will hurt performance even more.
> > > > For updates - we have vds_static thats hardly changed.
> > > > vds_statistics
> > > > that changes all the time. vds_dynamic is not changed allot -
> > but
> > > > is
> > > > updated all the time because of the status. I think it's best
> > to
> > > > split
> > > > it to the two existing tables (BTW - relevant for VM as well)
> > > >
> > > >
> > > > but we don't update it unless the status has changed, which is a
> > > > rare occurance?
> > > >
> > > > Actually - no. We can definitely see times we are updating vds_dynamic
> > > > with no reason at all. I tried to create patches for that - but it
> > > > happens from many different places in the code.
> > >
> > > what would be updated vds_dyanmic for status not originating in update
> > > run time info?
> >
> > We have separate DB flows for that (updateStatus and
> > updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
> > A question: do you know if we update status in updateVdsDynamic? :-) not
> > sure but I found a possible race for pending resources (cpu, mem), LOL :-)
> >
> > I think we do but not sure. Will check.

Of course it is, that was a rhetorical question :-) (a lot of emoticons and 
LOLs ;-))

> 
> 
> > Still holds my original thought for having vds_on_boot.
> >
> >
> >
> Let's talk f2f on Tuesday?

I'd prefer to reach conclusions here, I'd like everyone to be involved in a 
root issue like this one.

> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Kobi Ianko 
Joining in...
>From my point of view, in real life a user should have that many VDSs on one 
>Engine (from a DB point of view).
Modern DB system handles tables with millions of records and many relations, Do 
we really have a performance issue here?
We could prefer a more easy to maintain implantation in this case over DB 
performance


- Original Message -
> From: "Gilad Chaplik" 
> To: "Liran Zelkha" 
> Cc: de...@linode01.ovirt.org, "engine-devel" 
> Sent: Sunday, April 6, 2014 3:32:26 PM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> - Original Message -
> > From: "Liran Zelkha" 
> > To: "Gilad Chaplik" 
> > Cc: "Itamar Heim" , de...@linode01.ovirt.org,
> > "engine-devel" 
> > Sent: Sunday, April 6, 2014 3:26:24 PM
> > Subject: Re: [Engine-devel] vds_dynamic refactor
> > 
> > On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik  wrote:
> > 
> > > - Original Message -
> > > > From: "Itamar Heim" 
> > > > To: "Liran Zelkha" 
> > > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> > > "engine-devel" 
> > > > Sent: Sunday, April 6, 2014 11:33:12 AM
> > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > > >
> > > > On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > > > <mailto:ih...@redhat.com>> wrote:
> > > > >
> > > > > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> > > > >
> > > > > The problem is with both updates and selects.
> > > > > For selects - to get all the information for the VDS we have
> > > > > multiple
> > > > > joins. Adding another one will hurt performance even more.
> > > > > For updates - we have vds_static thats hardly changed.
> > > > > vds_statistics
> > > > > that changes all the time. vds_dynamic is not changed allot -
> > > but
> > > > > is
> > > > > updated all the time because of the status. I think it's best
> > > to
> > > > > split
> > > > > it to the two existing tables (BTW - relevant for VM as well)
> > > > >
> > > > >
> > > > > but we don't update it unless the status has changed, which is a
> > > > > rare occurance?
> > > > >
> > > > > Actually - no. We can definitely see times we are updating
> > > > > vds_dynamic
> > > > > with no reason at all. I tried to create patches for that - but it
> > > > > happens from many different places in the code.
> > > >
> > > > what would be updated vds_dyanmic for status not originating in update
> > > > run time info?
> > >
> > > We have separate DB flows for that (updateStatus and
> > > updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
> > > A question: do you know if we update status in updateVdsDynamic? :-) not
> > > sure but I found a possible race for pending resources (cpu, mem), LOL
> > > :-)
> > >
> > > I think we do but not sure. Will check.
> 
> Of course it is, that was a rhetorical question :-) (a lot of emoticons and
> LOLs ;-))
> 
> > 
> > 
> > > Still holds my original thought for having vds_on_boot.
> > >
> > >
> > >
> > Let's talk f2f on Tuesday?
> 
> I'd prefer to reach conclusions here, I'd like everyone to be involved in a
> root issue like this one.
> 
> > 
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Liran Zelkha 
On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:

> Joining in...
> From my point of view, in real life a user should have that many VDSs on
> one Engine (from a DB point of view).
> Modern DB system handles tables with millions of records and many
> relations, Do we really have a performance issue here?
> We could prefer a more easy to maintain implantation in this case over DB
> performance
>
> Yes we do. We make many queries on the VDS view, which is a VERY complex
view.

>
> - Original Message -
> > From: "Gilad Chaplik" 
> > To: "Liran Zelkha" 
> > Cc: de...@linode01.ovirt.org, "engine-devel" 
> > Sent: Sunday, April 6, 2014 3:32:26 PM
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> >
> > - Original Message -
> > > From: "Liran Zelkha" 
> > > To: "Gilad Chaplik" 
> > > Cc: "Itamar Heim" , de...@linode01.ovirt.org,
> > > "engine-devel" 
> > > Sent: Sunday, April 6, 2014 3:26:24 PM
> > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > >
> > > On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik 
> wrote:
> > >
> > > > - Original Message -
> > > > > From: "Itamar Heim" 
> > > > > To: "Liran Zelkha" 
> > > > > Cc: "Gilad Chaplik" ,
> de...@linode01.ovirt.org,
> > > > "engine-devel" 
> > > > > Sent: Sunday, April 6, 2014 11:33:12 AM
> > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > > > >
> > > > > On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > > > > <mailto:ih...@redhat.com>> wrote:
> > > > > >
> > > > > > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> > > > > >
> > > > > > The problem is with both updates and selects.
> > > > > > For selects - to get all the information for the VDS we
> have
> > > > > > multiple
> > > > > > joins. Adding another one will hurt performance even
> more.
> > > > > > For updates - we have vds_static thats hardly changed.
> > > > > > vds_statistics
> > > > > > that changes all the time. vds_dynamic is not changed
> allot -
> > > > but
> > > > > > is
> > > > > > updated all the time because of the status. I think it's
> best
> > > > to
> > > > > > split
> > > > > > it to the two existing tables (BTW - relevant for VM as
> well)
> > > > > >
> > > > > >
> > > > > > but we don't update it unless the status has changed, which
> is a
> > > > > > rare occurance?
> > > > > >
> > > > > > Actually - no. We can definitely see times we are updating
> > > > > > vds_dynamic
> > > > > > with no reason at all. I tried to create patches for that - but
> it
> > > > > > happens from many different places in the code.
> > > > >
> > > > > what would be updated vds_dyanmic for status not originating in
> update
> > > > > run time info?
> > > >
> > > > We have separate DB flows for that (updateStatus and
> > > > updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
> > > > A question: do you know if we update status in updateVdsDynamic? :-)
> not
> > > > sure but I found a possible race for pending resources (cpu, mem),
> LOL
> > > > :-)
> > > >
> > > > I think we do but not sure. Will check.
> >
> > Of course it is, that was a rhetorical question :-) (a lot of emoticons
> and
> > LOLs ;-))
> >
> > >
> > >
> > > > Still holds my original thought for having vds_on_boot.
> > > >
> > > >
> > > >
> > > Let's talk f2f on Tuesday?
> >
> > I'd prefer to reach conclusions here, I'd like everyone to be involved
> in a
> > root issue like this one.
>

What is the update frequency of this field?

> >
> > >
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> >
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Gilad Chaplik 
- Original Message -
> From: "Liran Zelkha" 
> To: "Kobi Ianko" 
> Cc: "Gilad Chaplik" , de...@linode01.ovirt.org, 
> "engine-devel" 
> Sent: Sunday, April 6, 2014 3:40:13 PM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:
> 
> > Joining in...
> > From my point of view, in real life a user should have that many VDSs on
> > one Engine (from a DB point of view).
> > Modern DB system handles tables with millions of records and many
> > relations, Do we really have a performance issue here?
> > We could prefer a more easy to maintain implantation in this case over DB
> > performance
> >
> > Yes we do. We make many queries on the VDS view, which is a VERY complex
> view.
> 

Actually I quite agree with Kobi, what is the plan for VMs? why do we start 
with VDS... 
what is the biggest deploy do you know of?

> >
> > - Original Message -
> > > From: "Gilad Chaplik" 
> > > To: "Liran Zelkha" 
> > > Cc: de...@linode01.ovirt.org, "engine-devel" 
> > > Sent: Sunday, April 6, 2014 3:32:26 PM
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > >
> > > - Original Message -
> > > > From: "Liran Zelkha" 
> > > > To: "Gilad Chaplik" 
> > > > Cc: "Itamar Heim" , de...@linode01.ovirt.org,
> > > > "engine-devel" 
> > > > Sent: Sunday, April 6, 2014 3:26:24 PM
> > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > > >
> > > > On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik 
> > wrote:
> > > >
> > > > > - Original Message -
> > > > > > From: "Itamar Heim" 
> > > > > > To: "Liran Zelkha" 
> > > > > > Cc: "Gilad Chaplik" ,
> > de...@linode01.ovirt.org,
> > > > > "engine-devel" 
> > > > > > Sent: Sunday, April 6, 2014 11:33:12 AM
> > > > > > Subject: Re: [Engine-devel] vds_dynamic refactor
> > > > > >
> > > > > > On 04/06/2014 11:32 AM, Liran Zelkha wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim  > > > > > > <mailto:ih...@redhat.com>> wrote:
> > > > > > >
> > > > > > > On 04/03/2014 07:51 PM, Liran Zelkha wrote:
> > > > > > >
> > > > > > > The problem is with both updates and selects.
> > > > > > > For selects - to get all the information for the VDS we
> > have
> > > > > > > multiple
> > > > > > > joins. Adding another one will hurt performance even
> > more.
> > > > > > > For updates - we have vds_static thats hardly changed.
> > > > > > > vds_statistics
> > > > > > > that changes all the time. vds_dynamic is not changed
> > allot -
> > > > > but
> > > > > > > is
> > > > > > > updated all the time because of the status. I think it's
> > best
> > > > > to
> > > > > > > split
> > > > > > > it to the two existing tables (BTW - relevant for VM as
> > well)
> > > > > > >
> > > > > > >
> > > > > > > but we don't update it unless the status has changed, which
> > is a
> > > > > > > rare occurance?
> > > > > > >
> > > > > > > Actually - no. We can definitely see times we are updating
> > > > > > > vds_dynamic
> > > > > > > with no reason at all. I tried to create patches for that - but
> > it
> > > > > > > happens from many different places in the code.
> > > > > >
> > > > > > what would be updated vds_dyanmic for status not originating in
> > update
> > > > > > run time info?
> > > > >
> > > > > We have separate DB flows for that (updateStatus and
> > > > > updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl).
> > > > > A question: do you know if we update status in updateVdsDynamic? :-)
> > not
> > > > > sure but I found a possible race for pending resources (cpu, mem),
> > LOL
> > > > > :-)
> > > > >
> > > > > I think we do but not sure. Will check.
> > >
> > > Of course it is, that was a rhetorical question :-) (a lot of emoticons
> > and
> > > LOLs ;-))
> > >
> > > >
> > > >
> > > > > Still holds my original thought for having vds_on_boot.
> > > > >
> > > > >
> > > > >
> > > > Let's talk f2f on Tuesday?
> > >
> > > I'd prefer to reach conclusions here, I'd like everyone to be involved
> > in a
> > > root issue like this one.
> >
> 
> What is the update frequency of this field?
> 

which field?
status? pending resources? on boot fields?
iinm, status is updated mostly by user actions, at least in positive scenarios, 
and not that often.


> > >
> > > >
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> > >
> >
> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Gilad Chaplik 
- Original Message -
> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Kobi Ianko" , de...@linode01.ovirt.org, "engine-devel" 
> 
> Sent: Sunday, April 6, 2014 8:51:02 PM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik  wrote:
> 
> > - Original Message -
> > > From: "Liran Zelkha" 
> > > To: "Kobi Ianko" 
> > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> > "engine-devel" 
> > > Sent: Sunday, April 6, 2014 3:40:13 PM
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > >
> > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:
> > >
> > > > Joining in...
> > > > From my point of view, in real life a user should have that many VDSs
> > on
> > > > one Engine (from a DB point of view).
> > > > Modern DB system handles tables with millions of records and many
> > > > relations, Do we really have a performance issue here?
> > > > We could prefer a more easy to maintain implantation in this case over
> > DB
> > > > performance
> > > >
> > > > Yes we do. We make many queries on the VDS view, which is a VERY
> > complex
> > > view.
> > >
> >
> > Actually I quite agree with Kobi, what is the plan for VMs? why do we
> > start with VDS...
> > what is the biggest deploy do you know of?
> >
> We start with VDS because in an idle system, with 200 hosts and several
> thousands VMs, this is what you get as the top queries against the
> database. Look at how many times getvds is called.
> [image: Inline image 1]
> BTW - the second query is an example of abusing the dynamic query
> mechanism. The 4th query (an update command) is a set of useless
> update_vds_dynamic commands.
> 
> For reference, the explain plan of get VDS is something like this:
> 
> QUERY PLAN
> 
> ---
>  Nested Loop  (cost=9.30..46.75 rows=6 width=9060) (actual
> time=0.063..0.068 rows=1 loops=1)
>Join Filter: (vds_static.vds_id = vds_statistics.vds_id)
>->  Seq Scan on vds_statistics  (cost=0.00..1.01 rows=1 width=109)
> (actual time=0.008..0.008 rows=1 loops=1)
>->  Nested Loop  (cost=9.30..45.64 rows=6 width=8983) (actual
> time=0.048..0.052 rows=1 loops=1)
>  Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id)
>  ->  Nested Loop Left Join  (cost=0.00..9.29 rows=1 width=1389)
> (actual time=0.013..0.013 rows=1 loops=1)
>->  Seq Scan on vds_groups  (cost=0.00..1.01 rows=1
> width=1271) (actual time=0.003..0.003 rows=1 loops=1)
>->  Index Scan using pk_storage_pool on storage_pool
>  (cost=0.00..8.27 rows=1 width=134) (actual time=0.008..0.008 rows=1
> loops=1)
>  Index Cond: (vds_groups.storage_pool_id = id)
>  ->  Hash Right Join  (cost=9.30..36.28 rows=6 width=7610) (actual
> time=0.033..0.037 rows=1 loops=1)
>Hash Cond: (vds_spm_id_map.vds_id = vds_static.vds_id)
>->  Seq Scan on vds_spm_id_map  (cost=0.00..22.30 rows=1230
> width=20) (actual time=0.003..0.003 rows=1 loops=1)
>->  Hash  (cost=9.29..9.29 rows=1 width=7606) (actual
> time=0.019..0.019 rows=1 loops=1)
>  Buckets: 1024  Batches: 1  Memory Usage: 2kB
>  ->  Nested Loop  (cost=0.00..9.29 rows=1 width=7606)
> (actual time=0.012..0.013 rows=1 loops=1)
>->  Seq Scan on vds_dynamic  (cost=0.00..1.01
> rows=1 width=1895) (actual time=0.006..0.006 rows=1 loops=1)
>->  Index Scan using pk_vds_static on vds_static
>  (cost=0.00..8.27 rows=1 width=5711) (actual time=0.005..0.006 rows=1
> loops=1)
>  Index Cond: (vds_id = vds_dynamic.vds_id)
>  Total runtime: 0.299 ms
> (19 rows)
> 
> It's terrible. Adding any additional join will make this worse. Please
> don't add any more tables...

Thank you for the detailed explanation, my comments:

* a very long time isn't an argument for not adding another table (should be 
neglectable);
currently we have an unrelated problem, we need to solve it.

* > We start with VDS because in an idle system, with 200 hosts and several
> thousands VMs, this is what you get as the top queries against the
> database.

so, if fetching VMs takes 10 minutes? and its get called a single time?

Re: [Devel] [Engine-devel] vds_dynamic refactor

2014-04-06 Thread Liran Zelkha 
On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik  wrote:

> - Original Message -
> > From: "Liran Zelkha" 
> > To: "Gilad Chaplik" 
> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> "engine-devel" 
> > Sent: Sunday, April 6, 2014 8:51:02 PM
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> >
> > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik 
> wrote:
> >
> > > - Original Message -
> > > > From: "Liran Zelkha" 
> > > > To: "Kobi Ianko" 
> > > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> > > "engine-devel" 
> > > > Sent: Sunday, April 6, 2014 3:40:13 PM
> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > >
> > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:
> > > >
> > > > > Joining in...
> > > > > From my point of view, in real life a user should have that many
> VDSs
> > > on
> > > > > one Engine (from a DB point of view).
> > > > > Modern DB system handles tables with millions of records and many
> > > > > relations, Do we really have a performance issue here?
> > > > > We could prefer a more easy to maintain implantation in this case
> over
> > > DB
> > > > > performance
> > > > >
> > > > > Yes we do. We make many queries on the VDS view, which is a VERY
> > > complex
> > > > view.
> > > >
> > >
> > > Actually I quite agree with Kobi, what is the plan for VMs? why do we
> > > start with VDS...
> > > what is the biggest deploy do you know of?
> > >
> > We start with VDS because in an idle system, with 200 hosts and several
> > thousands VMs, this is what you get as the top queries against the
> > database. Look at how many times getvds is called.
> > [image: Inline image 1]
> > BTW - the second query is an example of abusing the dynamic query
> > mechanism. The 4th query (an update command) is a set of useless
> > update_vds_dynamic commands.
> >
> > For reference, the explain plan of get VDS is something like this:
> >
> > QUERY PLAN
> >
> >
> ---
> >  Nested Loop  (cost=9.30..46.75 rows=6 width=9060) (actual
> > time=0.063..0.068 rows=1 loops=1)
> >Join Filter: (vds_static.vds_id = vds_statistics.vds_id)
> >->  Seq Scan on vds_statistics  (cost=0.00..1.01 rows=1 width=109)
> > (actual time=0.008..0.008 rows=1 loops=1)
> >->  Nested Loop  (cost=9.30..45.64 rows=6 width=8983) (actual
> > time=0.048..0.052 rows=1 loops=1)
> >  Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id)
> >  ->  Nested Loop Left Join  (cost=0.00..9.29 rows=1 width=1389)
> > (actual time=0.013..0.013 rows=1 loops=1)
> >->  Seq Scan on vds_groups  (cost=0.00..1.01 rows=1
> > width=1271) (actual time=0.003..0.003 rows=1 loops=1)
> >->  Index Scan using pk_storage_pool on storage_pool
> >  (cost=0.00..8.27 rows=1 width=134) (actual time=0.008..0.008 rows=1
> > loops=1)
> >  Index Cond: (vds_groups.storage_pool_id = id)
> >  ->  Hash Right Join  (cost=9.30..36.28 rows=6 width=7610)
> (actual
> > time=0.033..0.037 rows=1 loops=1)
> >Hash Cond: (vds_spm_id_map.vds_id = vds_static.vds_id)
> >->  Seq Scan on vds_spm_id_map  (cost=0.00..22.30
> rows=1230
> > width=20) (actual time=0.003..0.003 rows=1 loops=1)
> >->  Hash  (cost=9.29..9.29 rows=1 width=7606) (actual
> > time=0.019..0.019 rows=1 loops=1)
> >  Buckets: 1024  Batches: 1  Memory Usage: 2kB
> >  ->  Nested Loop  (cost=0.00..9.29 rows=1 width=7606)
> > (actual time=0.012..0.013 rows=1 loops=1)
> >->  Seq Scan on vds_dynamic  (cost=0.00..1.01
> > rows=1 width=1895) (actual time=0.006..0.006 rows=1 loops=1)
> >->  Index Scan using pk_vds_static on
> vds_static
> >  (cost=0.00..8.27 rows=1 width=5711) (actual time=0.005..0.006 rows=1
> > loops=1)
> >  Index Cond: (vds_id =
> vds_dynamic.vds_id)
> >  Total runtime: 0.299 ms
> > (19 rows)
> >
> > It's terrible. Adding any additional joi

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-09 Thread Yaniv Dary 
Why not move only status with changes a lot to statistics and leave everything 
as is? 

Yaniv 

- Original Message -

> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Kobi Ianko" , de...@linode01.ovirt.org, "engine-devel"
> 
> Sent: Monday, April 7, 2014 8:51:00 AM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor

> On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik < gchap...@redhat.com > wrote:

> > - Original Message -
> 
> > > From: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > To: "Gilad Chaplik" < gchap...@redhat.com >
> 
> > > Cc: "Kobi Ianko" < k...@redhat.com >, de...@linode01.ovirt.org ,
> > > "engine-devel" < engine-de...@ovirt.org >
> 
> > > Sent: Sunday, April 6, 2014 8:51:02 PM
> 
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> > >
> 
> > > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik < gchap...@redhat.com >
> > > wrote:
> 
> > >
> 
> > > > - Original Message -
> 
> > > > > From: "Liran Zelkha" < liran.zel...@gmail.com >
> 
> > > > > To: "Kobi Ianko" < k...@redhat.com >
> 
> > > > > Cc: "Gilad Chaplik" < gchap...@redhat.com >, de...@linode01.ovirt.org
> > > > > ,
> 
> > > > "engine-devel" < engine-de...@ovirt.org >
> 
> > > > > Sent: Sunday, April 6, 2014 3:40:13 PM
> 
> > > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> > > > >
> 
> > > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko < k...@redhat.com > wrote:
> 
> > > > >
> 
> > > > > > Joining in...
> 
> > > > > > From my point of view, in real life a user should have that many
> > > > > > VDSs
> 
> > > > on
> 
> > > > > > one Engine (from a DB point of view).
> 
> > > > > > Modern DB system handles tables with millions of records and many
> 
> > > > > > relations, Do we really have a performance issue here?
> 
> > > > > > We could prefer a more easy to maintain implantation in this case
> > > > > > over
> 
> > > > DB
> 
> > > > > > performance
> 
> > > > > >
> 
> > > > > > Yes we do. We make many queries on the VDS view, which is a VERY
> 
> > > > complex
> 
> > > > > view.
> 
> > > > >
> 
> > > >
> 
> > > > Actually I quite agree with Kobi, what is the plan for VMs? why do we
> 
> > > > start with VDS...
> 
> > > > what is the biggest deploy do you know of?
> 
> > > >
> 
> > > We start with VDS because in an idle system, with 200 hosts and several
> 
> > > thousands VMs, this is what you get as the top queries against the
> 
> > > database. Look at how many times getvds is called.
> 
> > > [image: Inline image 1]
> 
> > > BTW - the second query is an example of abusing the dynamic query
> 
> > > mechanism. The 4th query (an update command) is a set of useless
> 
> > > update_vds_dynamic commands.
> 
> > >
> 
> > > For reference, the explain plan of get VDS is something like this:
> 
> > >
> 
> > > QUERY PLAN
> 
> > >
> 
> > > ---
> 
> > > Nested Loop (cost=9.30..46.75 rows=6 width=9060) (actual
> 
> > > time=0.063..0.068 rows=1 loops=1)
> 
> > > Join Filter: (vds_static.vds_id = vds_statistics.vds_id)
> 
> > > -> Seq Scan on vds_statistics (cost=0.00..1.01 rows=1 width=109)
> 
> > > (actual time=0.008..0.008 rows=1 loops=1)
> 
> > > -> Nested Loop (cost=9.30..45.64 rows=6 width=8983) (actual
> 
> > > time=0.048..0.052 rows=1 loops=1)
> 
> > > Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id)
> 
> > > -> Nested Loop Left Join (cost=0.00..9.29 rows=1 width=1389)
> 
> > > (actual time=0.013..0.013 rows=1 loops=1)
> 
> > > -> Seq Scan on vds_groups (cost=0.00..1.01 rows=1
> 
> > > width=1271) (actual time=0.003..0.003 rows=1 loops=1)
> 
> > > -> Index Scan using pk_storage_pool on storage_pool
>

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-09 Thread Liran Zelkha 
On Wed, Apr 9, 2014 at 3:34 PM, Yaniv Dary  wrote:
>
> Why not move only status with changes a lot to statistics and leave 
> everything as is?
>
>
Exactly. No need for a new table. Use the existing ones.

>
>
> Yaniv
>
> 
>
> From: "Liran Zelkha" 
> To: "Gilad Chaplik" 
> Cc: "Kobi Ianko" , de...@linode01.ovirt.org, "engine-devel" 
> 
> Sent: Monday, April 7, 2014 8:51:00 AM
>
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
>
>
>
>
> On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik  wrote:
>>
>> - Original Message -
>> > From: "Liran Zelkha" 
>> > To: "Gilad Chaplik" 
>> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org, 
>> > "engine-devel" 
>> > Sent: Sunday, April 6, 2014 8:51:02 PM
>> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
>> >
>> > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik  wrote:
>> >
>> > > - Original Message -----
>> > > > From: "Liran Zelkha" 
>> > > > To: "Kobi Ianko" 
>> > > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
>> > > "engine-devel" 
>> > > > Sent: Sunday, April 6, 2014 3:40:13 PM
>> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
>> > > >
>> > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:
>> > > >
>> > > > > Joining in...
>> > > > > From my point of view, in real life a user should have that many VDSs
>> > > on
>> > > > > one Engine (from a DB point of view).
>> > > > > Modern DB system handles tables with millions of records and many
>> > > > > relations, Do we really have a performance issue here?
>> > > > > We could prefer a more easy to maintain implantation in this case 
>> > > > > over
>> > > DB
>> > > > > performance
>> > > > >
>> > > > > Yes we do. We make many queries on the VDS view, which is a VERY
>> > > complex
>> > > > view.
>> > > >
>> > >
>> > > Actually I quite agree with Kobi, what is the plan for VMs? why do we
>> > > start with VDS...
>> > > what is the biggest deploy do you know of?
>> > >
>> > We start with VDS because in an idle system, with 200 hosts and several
>> > thousands VMs, this is what you get as the top queries against the
>> > database. Look at how many times getvds is called.
>> > [image: Inline image 1]
>> > BTW - the second query is an example of abusing the dynamic query
>> > mechanism. The 4th query (an update command) is a set of useless
>> > update_vds_dynamic commands.
>> >
>> > For reference, the explain plan of get VDS is something like this:
>> >
>> > QUERY PLAN
>> >
>> > ---
>> >  Nested Loop  (cost=9.30..46.75 rows=6 width=9060) (actual
>> > time=0.063..0.068 rows=1 loops=1)
>> >Join Filter: (vds_static.vds_id = vds_statistics.vds_id)
>> >->  Seq Scan on vds_statistics  (cost=0.00..1.01 rows=1 width=109)
>> > (actual time=0.008..0.008 rows=1 loops=1)
>> >->  Nested Loop  (cost=9.30..45.64 rows=6 width=8983) (actual
>> > time=0.048..0.052 rows=1 loops=1)
>> >  Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id)
>> >  ->  Nested Loop Left Join  (cost=0.00..9.29 rows=1 width=1389)
>> > (actual time=0.013..0.013 rows=1 loops=1)
>> >->  Seq Scan on vds_groups  (cost=0.00..1.01 rows=1
>> > width=1271) (actual time=0.003..0.003 rows=1 loops=1)
>> >->  Index Scan using pk_storage_pool on storage_pool
>> >  (cost=0.00..8.27 rows=1 width=134) (actual time=0.008..0.008 rows=1
>> > loops=1)
>> >  Index Cond: (vds_groups.storage_pool_id = id)
>> >  ->  Hash Right Join  (cost=9.30..36.28 rows=6 width=7610) (actual
>> > time=0.033..0.037 rows=1 loops=1)
>> >Hash Cond: (vds_spm_id_map.vds_id = vds_static.vds_id)
>> >->  Seq Scan on vds_spm_id_map  (cost=0.00..22.30 rows=1230
>> > width=20) (actual time=0.003..0.003 rows=1 loops=1

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-10 Thread Gilad Chaplik 
- Original Message -
> From: "Liran Zelkha" 
> To: "Yaniv Dary" 
> Cc: "Gilad Chaplik" , "Kobi Ianko" , 
> de...@linode01.ovirt.org, "engine-devel"
> 
> Sent: Wednesday, April 9, 2014 4:22:39 PM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> On Wed, Apr 9, 2014 at 3:34 PM, Yaniv Dary  wrote:
> >
> > Why not move only status with changes a lot to statistics and leave
> > everything as is?
> >
> >
> Exactly. No need for a new table. Use the existing ones.

Why should the dwh read entire dynamic record for each status change/available 
resources change? (for VMs as well).

> 
> >
> >
> > Yaniv
> >
> > 
> >
> > From: "Liran Zelkha" 
> > To: "Gilad Chaplik" 
> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > "engine-devel" 
> > Sent: Monday, April 7, 2014 8:51:00 AM
> >
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> >
> >
> >
> >
> > On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik  wrote:
> >>
> >> - Original Message -
> >> > From: "Liran Zelkha" 
> >> > To: "Gilad Chaplik" 
> >> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> >> > "engine-devel" 
> >> > Sent: Sunday, April 6, 2014 8:51:02 PM
> >> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> >> >
> >> > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik 
> >> > wrote:
> >> >
> >> > > - Original Message -
> >> > > > From: "Liran Zelkha" 
> >> > > > To: "Kobi Ianko" 
> >> > > > Cc: "Gilad Chaplik" , de...@linode01.ovirt.org,
> >> > > "engine-devel" 
> >> > > > Sent: Sunday, April 6, 2014 3:40:13 PM
> >> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> >> > > >
> >> > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko  wrote:
> >> > > >
> >> > > > > Joining in...
> >> > > > > From my point of view, in real life a user should have that many
> >> > > > > VDSs
> >> > > on
> >> > > > > one Engine (from a DB point of view).
> >> > > > > Modern DB system handles tables with millions of records and many
> >> > > > > relations, Do we really have a performance issue here?
> >> > > > > We could prefer a more easy to maintain implantation in this case
> >> > > > > over
> >> > > DB
> >> > > > > performance
> >> > > > >
> >> > > > > Yes we do. We make many queries on the VDS view, which is a VERY
> >> > > complex
> >> > > > view.
> >> > > >
> >> > >
> >> > > Actually I quite agree with Kobi, what is the plan for VMs? why do we
> >> > > start with VDS...
> >> > > what is the biggest deploy do you know of?
> >> > >
> >> > We start with VDS because in an idle system, with 200 hosts and several
> >> > thousands VMs, this is what you get as the top queries against the
> >> > database. Look at how many times getvds is called.
> >> > [image: Inline image 1]
> >> > BTW - the second query is an example of abusing the dynamic query
> >> > mechanism. The 4th query (an update command) is a set of useless
> >> > update_vds_dynamic commands.
> >> >
> >> > For reference, the explain plan of get VDS is something like this:
> >> >
> >> > QUERY PLAN
> >> >
> >> > ---
> >> >  Nested Loop  (cost=9.30..46.75 rows=6 width=9060) (actual
> >> > time=0.063..0.068 rows=1 loops=1)
> >> >Join Filter: (vds_static.vds_id = vds_statistics.vds_id)
> >> >->  Seq Scan on vds_statistics  (cost=0.00..1.01 rows=1 width=109)
> >> > (actual time=0.008..0.008 rows=1 loops=1)
> >> >->  Nested Loop  (cost=9.30..45.64 rows=6 width=8983) (actual
> >> > time=0.048..0.052 rows=1 loops=1)
> >> >  Join Filter: (vds_groups.vds_group_id =
> >> >

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-10 Thread Yaniv Dary 


- Original Message -
> From: "Gilad Chaplik" 
> To: "Liran Zelkha" 
> Cc: "Yaniv Dary" , "Kobi Ianko" , 
> de...@linode01.ovirt.org, "engine-devel"
> 
> Sent: Thursday, April 10, 2014 11:07:12 AM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> - Original Message -
> > From: "Liran Zelkha" 
> > To: "Yaniv Dary" 
> > Cc: "Gilad Chaplik" , "Kobi Ianko" ,
> > de...@linode01.ovirt.org, "engine-devel"
> > 
> > Sent: Wednesday, April 9, 2014 4:22:39 PM
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > 
> > On Wed, Apr 9, 2014 at 3:34 PM, Yaniv Dary  wrote:
> > >
> > > Why not move only status with changes a lot to statistics and leave
> > > everything as is?
> > >
> > >
> > Exactly. No need for a new table. Use the existing ones.
> 
> Why should the dwh read entire dynamic record for each status
> change/available resources change? (for VMs as well).

The DWH doesn't work like that. We create views over dynamic\statistics and 
dynamic\static they go the the configuration and stats tables on the history db 
side.
>From dynamic\statistics we collect every minute no matter what and from 
>dynamic\static we collect only when update_date changes in static. 
We can not rely on the dynamic update_date since it changes so much, so from my 
point of view if we can cause minimal changes in dynamic and move status to 
statistics it would the best thing.
What we have currently is a task that run once a hour and checks if any change 
was made in the dynamic table (via very ugly joins and rejects) and sync is 
there was any change. 
It would even be better if you also move swap_size which doesn't change much to 
dynamic.

> 
> > 
> > >
> > >
> > > Yaniv
> > >
> > > 
> > >
> > > From: "Liran Zelkha" 
> > > To: "Gilad Chaplik" 
> > > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > > "engine-devel" 
> > > Sent: Monday, April 7, 2014 8:51:00 AM
> > >
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > >
> > >
> > >
> > >
> > > On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik 
> > > wrote:
> > >>
> > >> - Original Message -
> > >> > From: "Liran Zelkha" 
> > >> > To: "Gilad Chaplik" 
> > >> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > >> > "engine-devel" 
> > >> > Sent: Sunday, April 6, 2014 8:51:02 PM
> > >> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > >> >
> > >> > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik 
> > >> > wrote:
> > >> >
> > >> > > - Original Message -
> > >> > > > From: "Liran Zelkha" 
> > >> > > > To: "Kobi Ianko" 
> > >> > > > Cc: "Gilad Chaplik" ,
> > >> > > > de...@linode01.ovirt.org,
> > >> > > "engine-devel" 
> > >> > > > Sent: Sunday, April 6, 2014 3:40:13 PM
> > >> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > >> > > >
> > >> > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko 
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Joining in...
> > >> > > > > From my point of view, in real life a user should have that many
> > >> > > > > VDSs
> > >> > > on
> > >> > > > > one Engine (from a DB point of view).
> > >> > > > > Modern DB system handles tables with millions of records and
> > >> > > > > many
> > >> > > > > relations, Do we really have a performance issue here?
> > >> > > > > We could prefer a more easy to maintain implantation in this
> > >> > > > > case
> > >> > > > > over
> > >> > > DB
> > >> > > > > performance
> > >> > > > >
> > >> > > > > Yes we do. We make many queries on the VDS view, which is a VERY
> > >> > > complex
> > >> > > > view.
> > >> > > >
> > >> > >

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-10 Thread Gilad Chaplik 
- Original Message -
> From: "Yaniv Dary" 
> To: "Gilad Chaplik" 
> Cc: "Liran Zelkha" , "Kobi Ianko" , 
> de...@linode01.ovirt.org, "engine-devel"
> 
> Sent: Thursday, April 10, 2014 11:44:32 AM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> 
> 
> - Original Message -
> > From: "Gilad Chaplik" 
> > To: "Liran Zelkha" 
> > Cc: "Yaniv Dary" , "Kobi Ianko" ,
> > de...@linode01.ovirt.org, "engine-devel"
> > 
> > Sent: Thursday, April 10, 2014 11:07:12 AM
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > 
> > - Original Message -
> > > From: "Liran Zelkha" 
> > > To: "Yaniv Dary" 
> > > Cc: "Gilad Chaplik" , "Kobi Ianko"
> > > ,
> > > de...@linode01.ovirt.org, "engine-devel"
> > > 
> > > Sent: Wednesday, April 9, 2014 4:22:39 PM
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > 
> > > On Wed, Apr 9, 2014 at 3:34 PM, Yaniv Dary  wrote:
> > > >
> > > > Why not move only status with changes a lot to statistics and leave
> > > > everything as is?
> > > >
> > > >
> > > Exactly. No need for a new table. Use the existing ones.
> > 
> > Why should the dwh read entire dynamic record for each status
> > change/available resources change? (for VMs as well).
> 
> The DWH doesn't work like that. We create views over dynamic\statistics and
> dynamic\static they go the the configuration and stats tables on the history
> db side.
> From dynamic\statistics we collect every minute no matter what and from
> dynamic\static we collect only when update_date changes in static.

dynamic\statistics - dynamic\static

to fully understand it, we have 2 triggers to collect from dynamic?

> We can not rely on the dynamic update_date since it changes so much, so from
> my point of view if we can cause minimal changes in dynamic and move status
> to statistics it would the best thing.
> What we have currently is a task that run once a hour and checks if any
> change was made in the dynamic table (via very ugly joins and rejects) and
> sync is there was any change.
> It would even be better if you also move swap_size which doesn't change much
> to dynamic.
> 

oh... I thought that we had a trigger to dynamic (if ^ is correct we have).

+1 (out of 1) for you comment :-) and thanks Yaniv the elaborate explanation!

> > 
> > > 
> > > >
> > > >
> > > > Yaniv
> > > >
> > > > 
> > > >
> > > > From: "Liran Zelkha" 
> > > > To: "Gilad Chaplik" 
> > > > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > > > "engine-devel" 
> > > > Sent: Monday, April 7, 2014 8:51:00 AM
> > > >
> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik 
> > > > wrote:
> > > >>
> > > >> - Original Message -
> > > >> > From: "Liran Zelkha" 
> > > >> > To: "Gilad Chaplik" 
> > > >> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > > >> > "engine-devel" 
> > > >> > Sent: Sunday, April 6, 2014 8:51:02 PM
> > > >> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > >> >
> > > >> > On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik 
> > > >> > wrote:
> > > >> >
> > > >> > > - Original Message -
> > > >> > > > From: "Liran Zelkha" 
> > > >> > > > To: "Kobi Ianko" 
> > > >> > > > Cc: "Gilad Chaplik" ,
> > > >> > > > de...@linode01.ovirt.org,
> > > >> > > "engine-devel" 
> > > >> > > > Sent: Sunday, April 6, 2014 3:40:13 PM
> > > >> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > >> > > >
> > > >> > > > On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko 
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > &g

Re: [ovirt-devel] [Devel] [Engine-devel] vds_dynamic refactor

2014-04-10 Thread Yaniv Dary 


- Original Message -
> From: "Gilad Chaplik" 
> To: "Yaniv Dary" 
> Cc: "Liran Zelkha" , "Kobi Ianko" , 
> de...@linode01.ovirt.org, "engine-devel"
> 
> Sent: Thursday, April 10, 2014 12:30:52 PM
> Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> 
> - Original Message -
> > From: "Yaniv Dary" 
> > To: "Gilad Chaplik" 
> > Cc: "Liran Zelkha" , "Kobi Ianko"
> > , de...@linode01.ovirt.org, "engine-devel"
> > 
> > Sent: Thursday, April 10, 2014 11:44:32 AM
> > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > 
> > 
> > 
> > - Original Message -
> > > From: "Gilad Chaplik" 
> > > To: "Liran Zelkha" 
> > > Cc: "Yaniv Dary" , "Kobi Ianko" ,
> > > de...@linode01.ovirt.org, "engine-devel"
> > > 
> > > Sent: Thursday, April 10, 2014 11:07:12 AM
> > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > 
> > > - Original Message -
> > > > From: "Liran Zelkha" 
> > > > To: "Yaniv Dary" 
> > > > Cc: "Gilad Chaplik" , "Kobi Ianko"
> > > > ,
> > > > de...@linode01.ovirt.org, "engine-devel"
> > > > 
> > > > Sent: Wednesday, April 9, 2014 4:22:39 PM
> > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > > 
> > > > On Wed, Apr 9, 2014 at 3:34 PM, Yaniv Dary  wrote:
> > > > >
> > > > > Why not move only status with changes a lot to statistics and leave
> > > > > everything as is?
> > > > >
> > > > >
> > > > Exactly. No need for a new table. Use the existing ones.
> > > 
> > > Why should the dwh read entire dynamic record for each status
> > > change/available resources change? (for VMs as well).
> > 
> > The DWH doesn't work like that. We create views over dynamic\statistics and
> > dynamic\static they go the the configuration and stats tables on the
> > history
> > db side.
> > From dynamic\statistics we collect every minute no matter what and from
> > dynamic\static we collect only when update_date changes in static.
> 
> dynamic\statistics - dynamic\static
> 
> to fully understand it, we have 2 triggers to collect from dynamic?

No, the correct answer is that we don't have any triggers.
dynamic\statistics - we collect every minute for stats like status and cpu 
usage no matter what changed.
dynamic\static - we collect either when static table changes and update_date 
change or once an hour if dynamic changes, but we check this with join and 
rejects. This is because dynamic changes too much right now.
 
> 
> > We can not rely on the dynamic update_date since it changes so much, so
> > from
> > my point of view if we can cause minimal changes in dynamic and move status
> > to statistics it would the best thing.
> > What we have currently is a task that run once a hour and checks if any
> > change was made in the dynamic table (via very ugly joins and rejects) and
> > sync is there was any change.
> > It would even be better if you also move swap_size which doesn't change
> > much
> > to dynamic.
> > 
> 
> oh... I thought that we had a trigger to dynamic (if ^ is correct we have).
> 
> +1 (out of 1) for you comment :-) and thanks Yaniv the elaborate explanation!
> 
> > > 
> > > > 
> > > > >
> > > > >
> > > > > Yaniv
> > > > >
> > > > > 
> > > > >
> > > > > From: "Liran Zelkha" 
> > > > > To: "Gilad Chaplik" 
> > > > > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
> > > > > "engine-devel" 
> > > > > Sent: Monday, April 7, 2014 8:51:00 AM
> > > > >
> > > > > Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik 
> > > > > wrote:
> > > > >>
> > > > >> - Original Message -
> > > > >> > From: "Liran Zelkha" 
> > > > >> > To: "Gilad Chaplik" 
> > > > >> > Cc: "Kobi Ianko" , de...@linode01.ovirt.org,
&