Re: [ovirt-devel] New test failure on travis

[Edward Haas]

On Wed, Dec 7, 2016 at 11:54 PM, Nir Soffer  wrote:

> broken_on_ci is uses default name="OVIRT_CI", to mark it also for
> travis, we need another broken_on_ci with name="TRAVIS_CI".
>
> Maybe this test should run only if nm is active on the machine?
>

We need the test to always run when expected.
If NM is not running, the test will not run (silently) and we will never
know if there is a problem or not.

It is not convenient to mark each CI type as broken, why the test code
needs to know we have multiple
CI/s?

Currently, we run on CI tests that are not marked as 'functional'.
Perhaps we need another test type that can be mark not to run on simple CI.
"power-integration", "super-integration"?



>
>
> On Wed, Dec 7, 2016 at 11:23 PM, Dan Kenigsberg  wrote:
> > On Wed, Dec 7, 2016 at 2:03 PM, Nir Soffer  wrote:
> >> Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this:
> >
> > Odd, the code already has
> >
> > @broken_on_ci('NetworkManager should not be started on CI nodes')
> >
> >
> >>
> >> See https://travis-ci.org/oVirt/vdsm/jobs/181933329
> >>
> >> ==
> >>
> >> ERROR: test suite for  >> '/vdsm/tests/network/nmdbus_test.py'>
> >>
> >> --
> >>
> >> Traceback (most recent call last):
> >>
> >>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in
> run
> >>
> >> self.setUp()
> >>
> >>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in
> setUp
> >>
> >> self.setupContext(ancestor)
> >>
> >>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in
> >> setupContext
> >>
> >> try_run(context, names)
> >>
> >>   File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in
> try_run
> >>
> >> return func()
> >>
> >>   File "/vdsm/tests/testValidation.py", line 191, in wrapper
> >>
> >> return f(*args, **kwargs)
> >>
> >>   File "/vdsm/tests/testValidation.py", line 97, in wrapper
> >>
> >> return f(*args, **kwargs)
> >>
> >>   File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module
> >>
> >> NMDbus.init()
> >>
> >>   File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init
> >>
> >> NMDbus.bus = dbus.SystemBus()
> >>
> >>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194,
> in __new__
> >>
> >> private=private)
> >>
> >>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100,
> in __new__
> >>
> >> bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)
> >>
> >>   File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in
> __new__
> >>
> >> bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
> >>
> >> DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to
> >> connect to socket /var/run/dbus/system_bus_socket: No such file or
> >> directory
> >>
> >>  >> begin captured logging << 
> >>
> >> 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset
> >> --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None)
> >> (commands:69)
> >>
> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED:  =
> >> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
> >> (commands:93)
> >>
> >> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset
> >> --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None)
> >> (commands:69)
> >>
> >> 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED:  =
> >> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
> >> (commands:93)
> >>
> >> - >> end captured logging << -
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Review request for vdsm patch https://gerrit.ovirt.org/#/c/67912/

[Ramesh Nachimuthu]

Hi,

  I have sent the patch in subject to fix bz#1402248. It blocks gluster brick 
provisioning in oVirt 4.0.6. It will be helpful if someone can review this 
patch.

Regards,
Ramesh
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] New test failure on travis

[Nir Soffer]

broken_on_ci is uses default name="OVIRT_CI", to mark it also for
travis, we need another broken_on_ci with name="TRAVIS_CI".

Maybe this test should run only if nm is active on the machine?



On Wed, Dec 7, 2016 at 11:23 PM, Dan Kenigsberg  wrote:
> On Wed, Dec 7, 2016 at 2:03 PM, Nir Soffer  wrote:
>> Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this:
>
> Odd, the code already has
>
> @broken_on_ci('NetworkManager should not be started on CI nodes')
>
>
>>
>> See https://travis-ci.org/oVirt/vdsm/jobs/181933329
>>
>> ==
>>
>> ERROR: test suite for > '/vdsm/tests/network/nmdbus_test.py'>
>>
>> --
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run
>>
>> self.setUp()
>>
>>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
>>
>> self.setupContext(ancestor)
>>
>>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in
>> setupContext
>>
>> try_run(context, names)
>>
>>   File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
>>
>> return func()
>>
>>   File "/vdsm/tests/testValidation.py", line 191, in wrapper
>>
>> return f(*args, **kwargs)
>>
>>   File "/vdsm/tests/testValidation.py", line 97, in wrapper
>>
>> return f(*args, **kwargs)
>>
>>   File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module
>>
>> NMDbus.init()
>>
>>   File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init
>>
>> NMDbus.bus = dbus.SystemBus()
>>
>>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, in 
>> __new__
>>
>> private=private)
>>
>>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, in 
>> __new__
>>
>> bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)
>>
>>   File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in __new__
>>
>> bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
>>
>> DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to
>> connect to socket /var/run/dbus/system_bus_socket: No such file or
>> directory
>>
>>  >> begin captured logging << 
>>
>> 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset
>> --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None)
>> (commands:69)
>>
>> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED:  =
>> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
>> (commands:93)
>>
>> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset
>> --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None)
>> (commands:69)
>>
>> 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED:  =
>> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
>> (commands:93)
>>
>> - >> end captured logging << -
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] New test failure on travis

[Dan Kenigsberg]

On Wed, Dec 7, 2016 at 2:03 PM, Nir Soffer  wrote:
> Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this:

Odd, the code already has

@broken_on_ci('NetworkManager should not be started on CI nodes')


>
> See https://travis-ci.org/oVirt/vdsm/jobs/181933329
>
> ==
>
> ERROR: test suite for  '/vdsm/tests/network/nmdbus_test.py'>
>
> --
>
> Traceback (most recent call last):
>
>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run
>
> self.setUp()
>
>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
>
> self.setupContext(ancestor)
>
>   File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in
> setupContext
>
> try_run(context, names)
>
>   File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
>
> return func()
>
>   File "/vdsm/tests/testValidation.py", line 191, in wrapper
>
> return f(*args, **kwargs)
>
>   File "/vdsm/tests/testValidation.py", line 97, in wrapper
>
> return f(*args, **kwargs)
>
>   File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module
>
> NMDbus.init()
>
>   File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init
>
> NMDbus.bus = dbus.SystemBus()
>
>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, in 
> __new__
>
> private=private)
>
>   File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, in 
> __new__
>
> bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)
>
>   File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in __new__
>
> bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
>
> DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to
> connect to socket /var/run/dbus/system_bus_socket: No such file or
> directory
>
>  >> begin captured logging << 
>
> 2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset
> --cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None)
> (commands:69)
>
> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED:  =
> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
> (commands:93)
>
> 2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset
> --cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None)
> (commands:69)
>
> 2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED:  =
> 'Failed to get D-Bus connection: Operation not permitted\n';  = 1
> (commands:93)
>
> - >> end captured logging << -
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Eldad Marciano]

just forgot to mention that no customization required just plug & play he
will collect a large set of informative data by deafult

On Wed, Dec 7, 2016 at 10:54 PM, Eldad Marciano  wrote:

> In terms of measuring I used pgclu couple of times and it powerfull,easy
> to use, and provide very nice HTML reports
> http://pgcluu.darold.net/
>
> And also provide autovacum analysis
> http://pgcluu.darold.net/example/dolibarr-table-vacuums-analyzes.html
>
>
>
> On Wed, Dec 7, 2016 at 9:55 PM, Roy Golan  wrote:
>
>>
>>
>> On 7 December 2016 at 21:44, Roy Golan  wrote:
>>
>>>
>>>
>>> On 7 December 2016 at 21:00, Michal Skrivanek 
>>> wrote:
>>>


 On 07 Dec 2016, at 11:28, Yaniv Kaul  wrote:



 On Wed, Dec 7, 2016 at 10:57 AM, Roy Golan  wrote:

> Hi all,
>
> This is a discussion on the RFE[1] to provide a tool to perform full
> vacuum on our DBs.
>
> First if you are not familiar with vacuum please read this [2]
>
> # Backgroud
> ovirt 'engine' DB have several busy table with 2 differnt usage
> patten. One is audit_log and the others are the 'v*_statistics' tables and
> the difference between them is mostly inserts vs mostly hot updates.
> Tables with tons of updates creates garbage or 'dead' records that
> should be removed, and for this postgres have the aforementioned 
> autovacuum
> cleaner. It will make the db reuse its already allocated space to perform
> future updates/inserts and so on.
> Autovacuum is essential for a db to function optimally and tweaking it
> is out of the scope of the feature.
>
> Full vacuum is designed to reclaim the disk space and reset the table
> statistics. It is a heavy maintenance task, it takes an exclusive lock on
> the table and may take seconds to minutes. In some situations it is
> effectively a downtime due to the long table lock and should not be 
> running
> when the engine is running.
>

 So, effectively this should be interesting mostly/only for the audit
 log. All other busy table are mostly in-place updates

>>>
>>> Given that autovacuum is performing well the yes but if it starts to
>>> fall behind this may help a bit.
>>> audit_log is insert mostly and also delete, we remove a day, each day.
>>>


> # Critiria
> Provide a way to reclaim disk space claimed by the garbage created
> over time by the engine db and dwh.
>
> # Usage
> Either use it as part of the upgrade procedure (after all dbscipts
> execution)
>

 That does sound as a good start not requiring much user involvement

 or just provide the tool and admin will run in on demand
> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
> - invocation:
>  ```
>  tool: [dbname(default engine)] [table: (default all)]
>  ```
> - if we invoke it on upgrade than an installation plugin should be
> added to invoke with default, no interaction
>

 +1

 - since VACUUM ANALYZE is consider a recommended maintenance task we
> can to it by default and ask the user for FULL.
>

 When would you run it? ANALYZE nightly?

 No I'd still avoid doing this repeatedly, autovaccum should handle that
>>> as well, but this would cover situations where it isn't functioning
>>> optimally.
>>>
>>> I think  its worth adding a report of the db status and the rate of the
>>> autovacuum (a slight midifed version of the query mskrivanek ran on one of
>>> the production systems [3])  that will go to the logcollector. Perhaps the
>>> output of the ANALYZE will help as well.
>>>
>>> [3] https://gist.github.com/rgolangh/049cff30b89c5b29284ceee80a3
>>> 5dbb4#file-table_status_by_dead_rows-sql
>>>
>>
>>
>> Very interesting collection of pg scrips to measure bloat and vacuum -
>> needs access to postgres objects though
>>
>> - https://github.com/pgexperts/pgx_scripts
>> - https://github.com/pgexperts/pgx_scripts/blob/master/bloat/t
>> able_bloat_check.sql
>> - https://github.com/pgexperts/pgx_scripts/blob/master/vacuum/
>> last_autovacuum.sql
>>
>>
>>>
>>>
>
 Will the user know to answer intelligently if vacuum is needed or not?
 Except for 'yes, you need it', we cannot even provide a time estimate (I
 assume a disk space estimate is available!)

 perhaps we can estimate the bloat, there should be a github script to
>>> calculate that [4] not sure how good it is.
>>>
 I would suggest to run ANALYZE for sure and provide an option at the
 end of installation, to run the required command line - so make it as
 accessible as possible, but not part of the flow.


 If there are no significant gains why bother any other time but on
 upgrade when it can be run unconditionally?


 I'm wondering if 

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Eldad Marciano]

In terms of measuring I used pgclu couple of times and it powerfull,easy to
use, and provide very nice HTML reports
http://pgcluu.darold.net/

And also provide autovacum analysis
http://pgcluu.darold.net/example/dolibarr-table-vacuums-analyzes.html



On Wed, Dec 7, 2016 at 9:55 PM, Roy Golan  wrote:

>
>
> On 7 December 2016 at 21:44, Roy Golan  wrote:
>
>>
>>
>> On 7 December 2016 at 21:00, Michal Skrivanek 
>> wrote:
>>
>>>
>>>
>>> On 07 Dec 2016, at 11:28, Yaniv Kaul  wrote:
>>>
>>>
>>>
>>> On Wed, Dec 7, 2016 at 10:57 AM, Roy Golan  wrote:
>>>
 Hi all,

 This is a discussion on the RFE[1] to provide a tool to perform full
 vacuum on our DBs.

 First if you are not familiar with vacuum please read this [2]

 # Backgroud
 ovirt 'engine' DB have several busy table with 2 differnt usage patten.
 One is audit_log and the others are the 'v*_statistics' tables and the
 difference between them is mostly inserts vs mostly hot updates.
 Tables with tons of updates creates garbage or 'dead' records that
 should be removed, and for this postgres have the aforementioned autovacuum
 cleaner. It will make the db reuse its already allocated space to perform
 future updates/inserts and so on.
 Autovacuum is essential for a db to function optimally and tweaking it
 is out of the scope of the feature.

 Full vacuum is designed to reclaim the disk space and reset the table
 statistics. It is a heavy maintenance task, it takes an exclusive lock on
 the table and may take seconds to minutes. In some situations it is
 effectively a downtime due to the long table lock and should not be running
 when the engine is running.

>>>
>>> So, effectively this should be interesting mostly/only for the audit
>>> log. All other busy table are mostly in-place updates
>>>
>>
>> Given that autovacuum is performing well the yes but if it starts to fall
>> behind this may help a bit.
>> audit_log is insert mostly and also delete, we remove a day, each day.
>>
>>>
>>>
 # Critiria
 Provide a way to reclaim disk space claimed by the garbage created over
 time by the engine db and dwh.

 # Usage
 Either use it as part of the upgrade procedure (after all dbscipts
 execution)

>>>
>>> That does sound as a good start not requiring much user involvement
>>>
>>> or just provide the tool and admin will run in on demand
 - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
 - invocation:
  ```
  tool: [dbname(default engine)] [table: (default all)]
  ```
 - if we invoke it on upgrade than an installation plugin should be
 added to invoke with default, no interaction

>>>
>>> +1
>>>
>>> - since VACUUM ANALYZE is consider a recommended maintenance task we can
 to it by default and ask the user for FULL.

>>>
>>> When would you run it? ANALYZE nightly?
>>>
>>> No I'd still avoid doing this repeatedly, autovaccum should handle that
>> as well, but this would cover situations where it isn't functioning
>> optimally.
>>
>> I think  its worth adding a report of the db status and the rate of the
>> autovacuum (a slight midifed version of the query mskrivanek ran on one of
>> the production systems [3])  that will go to the logcollector. Perhaps the
>> output of the ANALYZE will help as well.
>>
>> [3] https://gist.github.com/rgolangh/049cff30b89c5b29284ceee80a3
>> 5dbb4#file-table_status_by_dead_rows-sql
>>
>
>
> Very interesting collection of pg scrips to measure bloat and vacuum -
> needs access to postgres objects though
>
> - https://github.com/pgexperts/pgx_scripts
> - https://github.com/pgexperts/pgx_scripts/blob/master/bloat/
> table_bloat_check.sql
> - https://github.com/pgexperts/pgx_scripts/blob/master/
> vacuum/last_autovacuum.sql
>
>
>>
>>

>>> Will the user know to answer intelligently if vacuum is needed or not?
>>> Except for 'yes, you need it', we cannot even provide a time estimate (I
>>> assume a disk space estimate is available!)
>>>
>>> perhaps we can estimate the bloat, there should be a github script to
>> calculate that [4] not sure how good it is.
>>
>>> I would suggest to run ANALYZE for sure and provide an option at the end
>>> of installation, to run the required command line - so make it as
>>> accessible as possible, but not part of the flow.
>>>
>>>
>>> If there are no significant gains why bother any other time but on
>>> upgrade when it can be run unconditionally?
>>>
>>>
>>> I'm wondering if the community can run ANALYZE on their database, and we
>>> can estimate how many are in dire need for full vacuum already.
>>> Y.
>>>
>>> I'll send a different mail for that.
>>
>>
>>>
>>> - remote db is supported as well, doesn't have to be local

>>>
>>> Well, not sure if we need to bother. It was introduced for large
>>> deployments where the host can't 

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Roy Golan]

On 7 December 2016 at 21:44, Roy Golan  wrote:

>
>
> On 7 December 2016 at 21:00, Michal Skrivanek  wrote:
>
>>
>>
>> On 07 Dec 2016, at 11:28, Yaniv Kaul  wrote:
>>
>>
>>
>> On Wed, Dec 7, 2016 at 10:57 AM, Roy Golan  wrote:
>>
>>> Hi all,
>>>
>>> This is a discussion on the RFE[1] to provide a tool to perform full
>>> vacuum on our DBs.
>>>
>>> First if you are not familiar with vacuum please read this [2]
>>>
>>> # Backgroud
>>> ovirt 'engine' DB have several busy table with 2 differnt usage patten.
>>> One is audit_log and the others are the 'v*_statistics' tables and the
>>> difference between them is mostly inserts vs mostly hot updates.
>>> Tables with tons of updates creates garbage or 'dead' records that
>>> should be removed, and for this postgres have the aforementioned autovacuum
>>> cleaner. It will make the db reuse its already allocated space to perform
>>> future updates/inserts and so on.
>>> Autovacuum is essential for a db to function optimally and tweaking it
>>> is out of the scope of the feature.
>>>
>>> Full vacuum is designed to reclaim the disk space and reset the table
>>> statistics. It is a heavy maintenance task, it takes an exclusive lock on
>>> the table and may take seconds to minutes. In some situations it is
>>> effectively a downtime due to the long table lock and should not be running
>>> when the engine is running.
>>>
>>
>> So, effectively this should be interesting mostly/only for the audit log.
>> All other busy table are mostly in-place updates
>>
>
> Given that autovacuum is performing well the yes but if it starts to fall
> behind this may help a bit.
> audit_log is insert mostly and also delete, we remove a day, each day.
>
>>
>>
>>> # Critiria
>>> Provide a way to reclaim disk space claimed by the garbage created over
>>> time by the engine db and dwh.
>>>
>>> # Usage
>>> Either use it as part of the upgrade procedure (after all dbscipts
>>> execution)
>>>
>>
>> That does sound as a good start not requiring much user involvement
>>
>> or just provide the tool and admin will run in on demand
>>> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
>>> - invocation:
>>>  ```
>>>  tool: [dbname(default engine)] [table: (default all)]
>>>  ```
>>> - if we invoke it on upgrade than an installation plugin should be added
>>> to invoke with default, no interaction
>>>
>>
>> +1
>>
>> - since VACUUM ANALYZE is consider a recommended maintenance task we can
>>> to it by default and ask the user for FULL.
>>>
>>
>> When would you run it? ANALYZE nightly?
>>
>> No I'd still avoid doing this repeatedly, autovaccum should handle that
> as well, but this would cover situations where it isn't functioning
> optimally.
>
> I think  its worth adding a report of the db status and the rate of the
> autovacuum (a slight midifed version of the query mskrivanek ran on one of
> the production systems [3])  that will go to the logcollector. Perhaps the
> output of the ANALYZE will help as well.
>
> [3] https://gist.github.com/rgolangh/049cff30b89c5b29284ceee80a35db
> b4#file-table_status_by_dead_rows-sql
>


Very interesting collection of pg scrips to measure bloat and vacuum -
needs access to postgres objects though

- https://github.com/pgexperts/pgx_scripts
-
https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql
-
https://github.com/pgexperts/pgx_scripts/blob/master/vacuum/last_autovacuum.sql


>
>
>>>
>> Will the user know to answer intelligently if vacuum is needed or not?
>> Except for 'yes, you need it', we cannot even provide a time estimate (I
>> assume a disk space estimate is available!)
>>
>> perhaps we can estimate the bloat, there should be a github script to
> calculate that [4] not sure how good it is.
>
>> I would suggest to run ANALYZE for sure and provide an option at the end
>> of installation, to run the required command line - so make it as
>> accessible as possible, but not part of the flow.
>>
>>
>> If there are no significant gains why bother any other time but on
>> upgrade when it can be run unconditionally?
>>
>>
>> I'm wondering if the community can run ANALYZE on their database, and we
>> can estimate how many are in dire need for full vacuum already.
>> Y.
>>
>> I'll send a different mail for that.
>
>
>>
>> - remote db is supported as well, doesn't have to be local
>>>
>>
>> Well, not sure if we need to bother. It was introduced for large
>> deployments where the host can't fit both engine and db load. Do we still
>> have this issue? I wouldn't say so for 4.1. It may be very niche case
>>
>> Running full vacuum is anyway a psql command, so there is no hidden cost
> here (to the development side I mean)
>
>
>> Thanks,
>> michal
>>
>>
>>> # Questions
>>>  - Will remote dwh have the credentials under
>>> /etc/ovirt-engine/engine.conf.d?
>>>  - Should  AAA schema be taken into account as well?
>>>
>>> Please review, thanks
>>> Roy
>>>
>>> [1] 

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Eyal Edri]

just saw you asked about lago job (its actually ost,  but nm) and not to
run it on your laptop via cli.

that's a bit more tricky but doable,  we are working on providing such
feature as part of the zuul project,  but maybe we can think on some temp
solution until it will be ready

On Dec 7, 2016 9:51 PM, wrote:

> yes,  iirc just give lago cli the url of the build in Jenkins instead of
> the repo url you use and it should work (at least repoman has this
> capability),  it might need some extra params.
>
> ping me tomorrow if it doesn't work for you
>
> On Dec 7, 2016 9:05 PM, "Piotr Kliczewski" 
> wrote:
>
>> Cool job, can we use those rpms in lago job?
>>
>> On Wed, Dec 7, 2016 at 7:13 PM, Oved Ourfali  wrote:
>> > Cool job indeed!
>> >
>> >
>> > On Dec 7, 2016 16:40, "Eyal Edri"  wrote:
>> >>
>> >> FYI,
>> >>
>> >> After hearing that this might be useful to many developers, I wanted to
>> >> bring to your attention the ovirt-engine master build job from patch
>> [1]
>> >> which allows you to build new rpms from an open ovirt-engine patch on
>> >> Gerrit.
>> >>
>> >> Its was created as temp job for a team who needed it a few months ago,
>> but
>> >> seems it might useful for other teams as well, so we decided to
>> publish it
>> >> even though its not standardized yet as part of our standard CI system.
>> >>
>> >> I hope you can find it useful for your teams.
>> >>
>> >>
>> >> [1]
>> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artif
>> acts-el7-x86_64_build_from_patch/
>> >>
>> >> --
>> >> Eyal Edri
>> >> Associate Manager
>> >> RHV DevOps
>> >> EMEA ENG Virtualization R
>> >> Red Hat Israel
>> >>
>> >> phone: +972-9-7692018
>> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>
>> >> ___
>> >> Devel mailing list
>> >> Devel@ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/devel
>> >
>> >
>> > ___
>> > Devel mailing list
>> > Devel@ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/devel
>>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Eyal Edri]

yes,  iirc just give lago cli the url of the build in Jenkins instead of
the repo url you use and it should work (at least repoman has this
capability),  it might need some extra params.

ping me tomorrow if it doesn't work for you

On Dec 7, 2016 9:05 PM, "Piotr Kliczewski" 
wrote:

> Cool job, can we use those rpms in lago job?
>
> On Wed, Dec 7, 2016 at 7:13 PM, Oved Ourfali  wrote:
> > Cool job indeed!
> >
> >
> > On Dec 7, 2016 16:40, "Eyal Edri"  wrote:
> >>
> >> FYI,
> >>
> >> After hearing that this might be useful to many developers, I wanted to
> >> bring to your attention the ovirt-engine master build job from patch [1]
> >> which allows you to build new rpms from an open ovirt-engine patch on
> >> Gerrit.
> >>
> >> Its was created as temp job for a team who needed it a few months ago,
> but
> >> seems it might useful for other teams as well, so we decided to publish
> it
> >> even though its not standardized yet as part of our standard CI system.
> >>
> >> I hope you can find it useful for your teams.
> >>
> >>
> >> [1]
> >> http://jenkins.ovirt.org/job/ovirt-engine_master_build-
> artifacts-el7-x86_64_build_from_patch/
> >>
> >> --
> >> Eyal Edri
> >> Associate Manager
> >> RHV DevOps
> >> EMEA ENG Virtualization R
> >> Red Hat Israel
> >>
> >> phone: +972-9-7692018
> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >> ___
> >> Devel mailing list
> >> Devel@ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/devel
> >
> >
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Roy Golan]

On 7 December 2016 at 21:00, Michal Skrivanek  wrote:

>
>
> On 07 Dec 2016, at 11:28, Yaniv Kaul  wrote:
>
>
>
> On Wed, Dec 7, 2016 at 10:57 AM, Roy Golan  wrote:
>
>> Hi all,
>>
>> This is a discussion on the RFE[1] to provide a tool to perform full
>> vacuum on our DBs.
>>
>> First if you are not familiar with vacuum please read this [2]
>>
>> # Backgroud
>> ovirt 'engine' DB have several busy table with 2 differnt usage patten.
>> One is audit_log and the others are the 'v*_statistics' tables and the
>> difference between them is mostly inserts vs mostly hot updates.
>> Tables with tons of updates creates garbage or 'dead' records that should
>> be removed, and for this postgres have the aforementioned autovacuum
>> cleaner. It will make the db reuse its already allocated space to perform
>> future updates/inserts and so on.
>> Autovacuum is essential for a db to function optimally and tweaking it is
>> out of the scope of the feature.
>>
>> Full vacuum is designed to reclaim the disk space and reset the table
>> statistics. It is a heavy maintenance task, it takes an exclusive lock on
>> the table and may take seconds to minutes. In some situations it is
>> effectively a downtime due to the long table lock and should not be running
>> when the engine is running.
>>
>
> So, effectively this should be interesting mostly/only for the audit log.
> All other busy table are mostly in-place updates
>

Given that autovacuum is performing well the yes but if it starts to fall
behind this may help a bit.
audit_log is insert mostly and also delete, we remove a day, each day.

>
>
>> # Critiria
>> Provide a way to reclaim disk space claimed by the garbage created over
>> time by the engine db and dwh.
>>
>> # Usage
>> Either use it as part of the upgrade procedure (after all dbscipts
>> execution)
>>
>
> That does sound as a good start not requiring much user involvement
>
> or just provide the tool and admin will run in on demand
>> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
>> - invocation:
>>  ```
>>  tool: [dbname(default engine)] [table: (default all)]
>>  ```
>> - if we invoke it on upgrade than an installation plugin should be added
>> to invoke with default, no interaction
>>
>
> +1
>
> - since VACUUM ANALYZE is consider a recommended maintenance task we can
>> to it by default and ask the user for FULL.
>>
>
> When would you run it? ANALYZE nightly?
>
> No I'd still avoid doing this repeatedly, autovaccum should handle that as
well, but this would cover situations where it isn't functioning optimally.

I think  its worth adding a report of the db status and the rate of the
autovacuum (a slight midifed version of the query mskrivanek ran on one of
the production systems [3])  that will go to the logcollector. Perhaps the
output of the ANALYZE will help as well.

[3]
https://gist.github.com/rgolangh/049cff30b89c5b29284ceee80a35dbb4#file-table_status_by_dead_rows-sql

>
>>
> Will the user know to answer intelligently if vacuum is needed or not?
> Except for 'yes, you need it', we cannot even provide a time estimate (I
> assume a disk space estimate is available!)
>
> perhaps we can estimate the bloat, there should be a github script to
calculate that [4] not sure how good it is.

> I would suggest to run ANALYZE for sure and provide an option at the end
> of installation, to run the required command line - so make it as
> accessible as possible, but not part of the flow.
>
>
> If there are no significant gains why bother any other time but on upgrade
> when it can be run unconditionally?
>
>
> I'm wondering if the community can run ANALYZE on their database, and we
> can estimate how many are in dire need for full vacuum already.
> Y.
>
> I'll send a different mail for that.


>
> - remote db is supported as well, doesn't have to be local
>>
>
> Well, not sure if we need to bother. It was introduced for large
> deployments where the host can't fit both engine and db load. Do we still
> have this issue? I wouldn't say so for 4.1. It may be very niche case
>
> Running full vacuum is anyway a psql command, so there is no hidden cost
here (to the development side I mean)


> Thanks,
> michal
>
>
>> # Questions
>>  - Will remote dwh have the credentials under
>> /etc/ovirt-engine/engine.conf.d?
>>  - Should  AAA schema be taken into account as well?
>>
>> Please review, thanks
>> Roy
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430
>> [2] https://www.postgresql.org/docs/9.2/static/runtime-config-
>> autovacuum.html
>> [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html
>>
>> ___
>> Users mailing list
>> us...@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
> ___
> Users mailing list
> us...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Nir Soffer]

On Wed, Dec 7, 2016 at 8:24 PM, Michal Skrivanek  wrote:
>
>
> On 07 Dec 2016, at 09:17, Oved Ourfali  wrote:
>
>
>
> On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:
>>
>> On 06/12/16 22:06 +0200, Arik Hadas wrote:
>>>
>>> Adam,
>>
>>
>> :)  You seem upset.  Sorry if I touched on a nerve...
>>
>>> Just out of curiosity: when you write "v2v has promised" - what exactly
>>> do you
>>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I
>>> that
>>> implemented the integration with virt-v2v? I'm not aware of such a
>>> promise by
>>> any of these options :)
>>
>>
>> Some history...
>>
>> Earlier this year Nir, Francesco (added), Shahar, and I began
>> discussing the similarities between what storage needed to do with
>> external commands and what was designed specifically for v2v.  I am
>> not sure if you were involved in the project at that time.  The plan
>> was to create common infrastructure that could be extended to fit the
>> unique needs of the verticals.  The v2v code was going to be moved
>> over to the new infrastructure (see [1]) and the only thing that
>> stopped the initial patch was lack of a VMWare testing environment for
>> verification.
>>
>> At that time storage refocused on developing verbs that used the new
>> infrastructure and have been maintaining its suitability for general
>> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
>> item and much more difficult now due to the early missed opportunity.
>>
>>> Anyway, let's say that you were given such a promise by someone and thus
>>> consider that mechanism to be deprecated - it doesn't really matter.
>>
>>
>> I may be biased but I think my opinion does matter.
>>
>>> The current implementation doesn't well fit to this flow (it requires
>>> per-volume job, it creates leases that are not needed for template's
>>> disks,
>>> ...) and with the "next-gen API" with proper support for virt flows not
>>> even
>>> being discussed with us (and iiuc also not with the infra team) yet, I
>>> don't
>>> understand what do you suggest except for some strong, though irrelevant,
>>> statements.
>>
>>
>> If you are willing to engage in a good-faith technical discussion I am
>> sure I can help you to understand.  These operations to storage demand
>> some form of locking protection.  If volume leases aren't appropriate then
>> perhaps we should use the VM Leases / xleases that Nir is finishing
>> off for 4.1 now.
>>
>>> I suggest loud and clear to reuse (not to add dependencies, not to
>>> enhance, ..)
>>> an existing mechanism for a very similar flow of virt-v2v that works well
>>> and
>>> simple.
>>
>>
>> I clearly remember discussions involving infra (hello Oved), virt
>> (hola Michal), and storage where we decided that new APIs performing
>> async operations involving external commands should use the HostJobs
>> infrastructure instead of adding more information to Host Stats.
>> These were the "famous" entity polling meetings.
>>
>> Of course plans can change but I have never been looped into any such
>> discussions.
>>
>
> Well, I think that when someone builds a good infrastructure he first needs
> to talk to all consumers and make sure it fits.
> In this case it seems like most work was done to fit the storage use-case,
> and now you check whether it can fit others as well
>
> IMO it makes much more sense to use events where possible (and you've
> promised to use those as well, but I don't see you doing that...). v2v
> should use events for sure, and they have promised to do that in the past,
> instead of using the v2v jobs. The reason events weren't used originally
> with the v2v feature, was that it was too risky and the events
> infrastructure was added too late in the game.
>
>
> Revisiting and refactoring code which is already in use is always a bit of
> luxury we can rarely prioritize.
> So indeed v2v is not using events. The
> generalization work has been done to some extent, but there is no incentive
> to rewrite it completely.

On the vdsm side we don't need to rewrite the interesting parts, just run the
code using the current infrastructure.

The first step is to use the jobs framework and delete the stale copy
of the jobs
framework in v2v.py.

Then we can easily schedule the jobs in the task manager instead of starting
a thread for each job - one line to schedule a job in hsm - see the sparsify
patch.

I think Shahar started to work on it, we need to rebase his patch and finish it.

> On the other hand we are now trying to add events to migration progress
> reporting and hand over since that area is being touched due to post-copy
> enhancements.
> So, when there is a practical chance to improve functionality by utilizing
> events it indeed should be the first choice
>
>
>
>>>
>>> Do you "promise" to implement your "next gen API" for 4.1 as an
>>> alternative?
>>
>>
>> I guess we need the design first.
>>
>>
>>> On Tue, Dec 6, 2016 at 

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Piotr Kliczewski]

Cool job, can we use those rpms in lago job?

On Wed, Dec 7, 2016 at 7:13 PM, Oved Ourfali  wrote:
> Cool job indeed!
>
>
> On Dec 7, 2016 16:40, "Eyal Edri"  wrote:
>>
>> FYI,
>>
>> After hearing that this might be useful to many developers, I wanted to
>> bring to your attention the ovirt-engine master build job from patch [1]
>> which allows you to build new rpms from an open ovirt-engine patch on
>> Gerrit.
>>
>> Its was created as temp job for a team who needed it a few months ago, but
>> seems it might useful for other teams as well, so we decided to publish it
>> even though its not standardized yet as part of our standard CI system.
>>
>> I hope you can find it useful for your teams.
>>
>>
>> [1]
>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64_build_from_patch/
>>
>> --
>> Eyal Edri
>> Associate Manager
>> RHV DevOps
>> EMEA ENG Virtualization R
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Oved Ourfali]

On Dec 7, 2016 20:24, "Michal Skrivanek"  wrote:
>
>
>
> On 07 Dec 2016, at 09:17, Oved Ourfali  wrote:
>
>>
>>
>> On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:
>>>
>>> On 06/12/16 22:06 +0200, Arik Hadas wrote:

 Adam,
>>>
>>>
>>> :)  You seem upset.  Sorry if I touched on a nerve...
>>>
 Just out of curiosity: when you write "v2v has promised" - what
exactly do you
 mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and
I that
 implemented the integration with virt-v2v? I'm not aware of such a
promise by
 any of these options :)
>>>
>>>
>>> Some history...
>>>
>>> Earlier this year Nir, Francesco (added), Shahar, and I began
>>> discussing the similarities between what storage needed to do with
>>> external commands and what was designed specifically for v2v.  I am
>>> not sure if you were involved in the project at that time.  The plan
>>> was to create common infrastructure that could be extended to fit the
>>> unique needs of the verticals.  The v2v code was going to be moved
>>> over to the new infrastructure (see [1]) and the only thing that
>>> stopped the initial patch was lack of a VMWare testing environment for
>>> verification.
>>>
>>> At that time storage refocused on developing verbs that used the new
>>> infrastructure and have been maintaining its suitability for general
>>> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
>>> item and much more difficult now due to the early missed opportunity.
>>>
 Anyway, let's say that you were given such a promise by someone and
thus
 consider that mechanism to be deprecated - it doesn't really matter.
>>>
>>>
>>> I may be biased but I think my opinion does matter.
>>>
 The current implementation doesn't well fit to this flow (it requires
 per-volume job, it creates leases that are not needed for template's
disks,
 ...) and with the "next-gen API" with proper support for virt flows
not even
 being discussed with us (and iiuc also not with the infra team) yet, I
don't
 understand what do you suggest except for some strong, though
irrelevant,
 statements.
>>>
>>>
>>> If you are willing to engage in a good-faith technical discussion I am
>>> sure I can help you to understand.  These operations to storage demand
>>> some form of locking protection.  If volume leases aren't appropriate
then
>>> perhaps we should use the VM Leases / xleases that Nir is finishing
>>> off for 4.1 now.
>>>
 I suggest loud and clear to reuse (not to add dependencies, not to
enhance, ..)
 an existing mechanism for a very similar flow of virt-v2v that works
well and
 simple.
>>>
>>>
>>> I clearly remember discussions involving infra (hello Oved), virt
>>> (hola Michal), and storage where we decided that new APIs performing
>>> async operations involving external commands should use the HostJobs
>>> infrastructure instead of adding more information to Host Stats.
>>> These were the "famous" entity polling meetings.
>>>
>>> Of course plans can change but I have never been looped into any such
>>> discussions.
>>>
>>
>> Well, I think that when someone builds a good infrastructure he first
needs to talk to all consumers and make sure it fits.
>> In this case it seems like most work was done to fit the storage
use-case, and now you check whether it can fit others as well
>>
>> IMO it makes much more sense to use events where possible (and you've
promised to use those as well, but I don't see you doing that...). v2v
should use events for sure, and they have promised to do that in the past,
instead of using the v2v jobs. The reason events weren't used originally
with the v2v feature, was that it was too risky and the events
infrastructure was added too late in the game.
>
>
> Revisiting and refactoring code which is already in use is always a bit
of luxury we can rarely prioritize. So indeed v2v is not using events. The
generalization work has been done to some extent, but there is no incentive
to rewrite it completely.
> On the other hand we are now trying to add events to migration progress
reporting and hand over since that area is being touched due to post-copy
enhancements.
> So, when there is a practical chance to improve functionality by
utilizing events it indeed should be the first choice

+1.
Well understood.

>
>>
>>

 Do you "promise" to implement your "next gen API" for 4.1 as an
alternative?
>>>
>>>
>>> I guess we need the design first.
>>>
>>>
 On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke  wrote:

On 05/12/16 11:17 +0200, Arik Hadas wrote:



On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer 
wrote:

   On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud <
smela...@redhat.com>
wrote:
   >
   > Hi!
   >
   > I'm currently working on integration of 

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Michal Skrivanek]

On 07 Dec 2016, at 09:17, Oved Ourfali  wrote:



On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:

> On 06/12/16 22:06 +0200, Arik Hadas wrote:
>
>> Adam,
>>
>
> :)  You seem upset.  Sorry if I touched on a nerve...
>
> Just out of curiosity: when you write "v2v has promised" - what exactly do
>> you
>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I
>> that
>> implemented the integration with virt-v2v? I'm not aware of such a
>> promise by
>> any of these options :)
>>
>
> Some history...
>
> Earlier this year Nir, Francesco (added), Shahar, and I began
> discussing the similarities between what storage needed to do with
> external commands and what was designed specifically for v2v.  I am
> not sure if you were involved in the project at that time.  The plan
> was to create common infrastructure that could be extended to fit the
> unique needs of the verticals.  The v2v code was going to be moved
> over to the new infrastructure (see [1]) and the only thing that
> stopped the initial patch was lack of a VMWare testing environment for
> verification.
>
> At that time storage refocused on developing verbs that used the new
> infrastructure and have been maintaining its suitability for general
> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
> item and much more difficult now due to the early missed opportunity.
>
> Anyway, let's say that you were given such a promise by someone and thus
>> consider that mechanism to be deprecated - it doesn't really matter.
>>
>
> I may be biased but I think my opinion does matter.
>
> The current implementation doesn't well fit to this flow (it requires
>> per-volume job, it creates leases that are not needed for template's
>> disks,
>> ...) and with the "next-gen API" with proper support for virt flows not
>> even
>> being discussed with us (and iiuc also not with the infra team) yet, I
>> don't
>> understand what do you suggest except for some strong, though irrelevant,
>> statements.
>>
>
> If you are willing to engage in a good-faith technical discussion I am
> sure I can help you to understand.  These operations to storage demand
> some form of locking protection.  If volume leases aren't appropriate then
> perhaps we should use the VM Leases / xleases that Nir is finishing
> off for 4.1 now.
>
> I suggest loud and clear to reuse (not to add dependencies, not to
>> enhance, ..)
>> an existing mechanism for a very similar flow of virt-v2v that works well
>> and
>> simple.
>>
>
> I clearly remember discussions involving infra (hello Oved), virt
> (hola Michal), and storage where we decided that new APIs performing
> async operations involving external commands should use the HostJobs
> infrastructure instead of adding more information to Host Stats.
> These were the "famous" entity polling meetings.
>
> Of course plans can change but I have never been looped into any such
> discussions.
>
>
Well, I think that when someone builds a good infrastructure he first needs
to talk to all consumers and make sure it fits.
In this case it seems like most work was done to fit the storage use-case,
and now you check whether it can fit others as well

IMO it makes much more sense to use events where possible (and you've
promised to use those as well, but I don't see you doing that...). v2v
should use events for sure, and they have promised to do that in the past,
instead of using the v2v jobs. The reason events weren't used originally
with the v2v feature, was that it was too risky and the events
infrastructure was added too late in the game.


Revisiting and refactoring code which is already in use is always a bit of
luxury we can rarely prioritize. So indeed v2v is not using events. The
generalization work has been done to some extent, but there is no incentive
to rewrite it completely.
On the other hand we are now trying to add events to migration progress
reporting and hand over since that area is being touched due to post-copy
enhancements.
So, when there is a practical chance to improve functionality by utilizing
events it indeed should be the first choice




> Do you "promise" to implement your "next gen API" for 4.1 as an
>> alternative?
>>
>
> I guess we need the design first.
>
>
> On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke  wrote:
>>
>>On 05/12/16 11:17 +0200, Arik Hadas wrote:
>>
>>
>>
>>On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer 
>> wrote:
>>
>>   On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud <
>> smela...@redhat.com>
>>wrote:
>>   >
>>   > Hi!
>>   >
>>   > I'm currently working on integration of virt-sysprep into
>> oVirt.
>>   >
>>   > Usually, if user creates a template from a regular VM, and
>> then
>>creates
>>   new VMs from this template, these new VMs inherit all
>> configuration
>>of the
>>   original VM, including SSH 

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Oved Ourfali]

On Dec 7, 2016 20:16, "Nir Soffer"  wrote:
>
> On Wed, Dec 7, 2016 at 8:10 PM, Oved Ourfali  wrote:
> > On Dec 7, 2016 16:00, "Nir Soffer"  wrote:
> >>
> >> On Wed, Dec 7, 2016 at 10:17 AM, Oved Ourfali 
wrote:
> >> >
> >> >
> >> > On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke 
wrote:
> >> >>
> >> >> On 06/12/16 22:06 +0200, Arik Hadas wrote:
> >> >>>
> >> >>> Adam,
> >> >>
> >> >>
> >> >> :)  You seem upset.  Sorry if I touched on a nerve...
> >> >>
> >> >>> Just out of curiosity: when you write "v2v has promised" - what
> >> >>> exactly
> >> >>> do you
> >> >>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar
and
> >> >>> I
> >> >>> that
> >> >>> implemented the integration with virt-v2v? I'm not aware of such a
> >> >>> promise by
> >> >>> any of these options :)
> >> >>
> >> >>
> >> >> Some history...
> >> >>
> >> >> Earlier this year Nir, Francesco (added), Shahar, and I began
> >> >> discussing the similarities between what storage needed to do with
> >> >> external commands and what was designed specifically for v2v.  I am
> >> >> not sure if you were involved in the project at that time.  The plan
> >> >> was to create common infrastructure that could be extended to fit
the
> >> >> unique needs of the verticals.  The v2v code was going to be moved
> >> >> over to the new infrastructure (see [1]) and the only thing that
> >> >> stopped the initial patch was lack of a VMWare testing environment
for
> >> >> verification.
> >> >>
> >> >> At that time storage refocused on developing verbs that used the new
> >> >> infrastructure and have been maintaining its suitability for general
> >> >> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
> >> >> item and much more difficult now due to the early missed
opportunity.
> >> >>
> >> >>> Anyway, let's say that you were given such a promise by someone and
> >> >>> thus
> >> >>> consider that mechanism to be deprecated - it doesn't really
matter.
> >> >>
> >> >>
> >> >> I may be biased but I think my opinion does matter.
> >> >>
> >> >>> The current implementation doesn't well fit to this flow (it
requires
> >> >>> per-volume job, it creates leases that are not needed for
template's
> >> >>> disks,
> >> >>> ...) and with the "next-gen API" with proper support for virt flows
> >> >>> not
> >> >>> even
> >> >>> being discussed with us (and iiuc also not with the infra team)
yet, I
> >> >>> don't
> >> >>> understand what do you suggest except for some strong, though
> >> >>> irrelevant,
> >> >>> statements.
> >> >>
> >> >>
> >> >> If you are willing to engage in a good-faith technical discussion I
am
> >> >> sure I can help you to understand.  These operations to storage
demand
> >> >> some form of locking protection.  If volume leases aren't
appropriate
> >> >> then
> >> >> perhaps we should use the VM Leases / xleases that Nir is finishing
> >> >> off for 4.1 now.
> >> >>
> >> >>> I suggest loud and clear to reuse (not to add dependencies, not to
> >> >>> enhance, ..)
> >> >>> an existing mechanism for a very similar flow of virt-v2v that
works
> >> >>> well
> >> >>> and
> >> >>> simple.
> >> >>
> >> >>
> >> >> I clearly remember discussions involving infra (hello Oved), virt
> >> >> (hola Michal), and storage where we decided that new APIs performing
> >> >> async operations involving external commands should use the HostJobs
> >> >> infrastructure instead of adding more information to Host Stats.
> >> >> These were the "famous" entity polling meetings.
> >>
> >> We discussed these issues behind close doors, not in the public mailing
> >> list,
> >> so it is not surprising that people do not know about the agreements we
> >> had.
> >>
> >
> > The core team was there. So it is surprising.
> >
> >> >>
> >> >> Of course plans can change but I have never been looped into any
such
> >> >> discussions.
> >> >>
> >> >
> >> > Well, I think that when someone builds a good infrastructure he first
> >> > needs
> >> > to talk to all consumers and make sure it fits.
> >> > In this case it seems like most work was done to fit the storage
> >> > use-case,
> >> > and now you check whether it can fit others as well
> >>
> >> The jobs framework is generic and can be used for any subsystem,
> >> there is nothing related to storage about it. But modifying disks *is*
> >> a storage operation, even if someone from the virt team worked on it.
> >>
> >> V2v is also storage operation - if we compare it with copying disks:
> >>
> >> - we create a new volume that nobody is using yet
> >> - if the operation fails, the disk must be in illegal state
> >> - if the operation fails we delete the disks
> >> - if the operation succeeds the volume must be legal
> >> - we need to limit the number of operations on a host
> >> - we need to detect the job state if the host becomes non-responsive
> >> - we may want to fence the job if the host becomes 

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Nir Soffer]

On Wed, Dec 7, 2016 at 8:10 PM, Oved Ourfali  wrote:
> On Dec 7, 2016 16:00, "Nir Soffer"  wrote:
>>
>> On Wed, Dec 7, 2016 at 10:17 AM, Oved Ourfali  wrote:
>> >
>> >
>> > On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:
>> >>
>> >> On 06/12/16 22:06 +0200, Arik Hadas wrote:
>> >>>
>> >>> Adam,
>> >>
>> >>
>> >> :)  You seem upset.  Sorry if I touched on a nerve...
>> >>
>> >>> Just out of curiosity: when you write "v2v has promised" - what
>> >>> exactly
>> >>> do you
>> >>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and
>> >>> I
>> >>> that
>> >>> implemented the integration with virt-v2v? I'm not aware of such a
>> >>> promise by
>> >>> any of these options :)
>> >>
>> >>
>> >> Some history...
>> >>
>> >> Earlier this year Nir, Francesco (added), Shahar, and I began
>> >> discussing the similarities between what storage needed to do with
>> >> external commands and what was designed specifically for v2v.  I am
>> >> not sure if you were involved in the project at that time.  The plan
>> >> was to create common infrastructure that could be extended to fit the
>> >> unique needs of the verticals.  The v2v code was going to be moved
>> >> over to the new infrastructure (see [1]) and the only thing that
>> >> stopped the initial patch was lack of a VMWare testing environment for
>> >> verification.
>> >>
>> >> At that time storage refocused on developing verbs that used the new
>> >> infrastructure and have been maintaining its suitability for general
>> >> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
>> >> item and much more difficult now due to the early missed opportunity.
>> >>
>> >>> Anyway, let's say that you were given such a promise by someone and
>> >>> thus
>> >>> consider that mechanism to be deprecated - it doesn't really matter.
>> >>
>> >>
>> >> I may be biased but I think my opinion does matter.
>> >>
>> >>> The current implementation doesn't well fit to this flow (it requires
>> >>> per-volume job, it creates leases that are not needed for template's
>> >>> disks,
>> >>> ...) and with the "next-gen API" with proper support for virt flows
>> >>> not
>> >>> even
>> >>> being discussed with us (and iiuc also not with the infra team) yet, I
>> >>> don't
>> >>> understand what do you suggest except for some strong, though
>> >>> irrelevant,
>> >>> statements.
>> >>
>> >>
>> >> If you are willing to engage in a good-faith technical discussion I am
>> >> sure I can help you to understand.  These operations to storage demand
>> >> some form of locking protection.  If volume leases aren't appropriate
>> >> then
>> >> perhaps we should use the VM Leases / xleases that Nir is finishing
>> >> off for 4.1 now.
>> >>
>> >>> I suggest loud and clear to reuse (not to add dependencies, not to
>> >>> enhance, ..)
>> >>> an existing mechanism for a very similar flow of virt-v2v that works
>> >>> well
>> >>> and
>> >>> simple.
>> >>
>> >>
>> >> I clearly remember discussions involving infra (hello Oved), virt
>> >> (hola Michal), and storage where we decided that new APIs performing
>> >> async operations involving external commands should use the HostJobs
>> >> infrastructure instead of adding more information to Host Stats.
>> >> These were the "famous" entity polling meetings.
>>
>> We discussed these issues behind close doors, not in the public mailing
>> list,
>> so it is not surprising that people do not know about the agreements we
>> had.
>>
>
> The core team was there. So it is surprising.
>
>> >>
>> >> Of course plans can change but I have never been looped into any such
>> >> discussions.
>> >>
>> >
>> > Well, I think that when someone builds a good infrastructure he first
>> > needs
>> > to talk to all consumers and make sure it fits.
>> > In this case it seems like most work was done to fit the storage
>> > use-case,
>> > and now you check whether it can fit others as well
>>
>> The jobs framework is generic and can be used for any subsystem,
>> there is nothing related to storage about it. But modifying disks *is*
>> a storage operation, even if someone from the virt team worked on it.
>>
>> V2v is also storage operation - if we compare it with copying disks:
>>
>> - we create a new volume that nobody is using yet
>> - if the operation fails, the disk must be in illegal state
>> - if the operation fails we delete the disks
>> - if the operation succeeds the volume must be legal
>> - we need to limit the number of operations on a host
>> - we need to detect the job state if the host becomes non-responsive
>> - we may want to fence the job if the host becomes non-responsive
>>   in volume jobs, we can increment the volume generation and run
>>   the same job on another host.
>> - we want to take a lease on storage to ensure that other hosts cannot
>>   access the same entity, or that the job will fail if someone else is
>> using
>>   this entity
>> - we want to take 

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Oved Ourfali]

Cool job indeed!

On Dec 7, 2016 16:40, "Eyal Edri"  wrote:

> FYI,
>
> After hearing that this might be useful to many developers, I wanted to
> bring to your attention the ovirt-engine master build job from patch [1]
> which allows you to build new rpms from an open ovirt-engine patch on
> Gerrit.
>
> Its was created as temp job for a team who needed it a few months ago, but
> seems it might useful for other teams as well, so we decided to publish it
> even though its not standardized yet as part of our standard CI system.
>
> I hope you can find it useful for your teams.
>
>
> [1] http://jenkins.ovirt.org/job/ovirt-engine_master_build-
> artifacts-el7-x86_64_build_from_patch/
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Oved Ourfali]

On Dec 7, 2016 16:00, "Nir Soffer"  wrote:
>
> On Wed, Dec 7, 2016 at 10:17 AM, Oved Ourfali  wrote:
> >
> >
> > On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:
> >>
> >> On 06/12/16 22:06 +0200, Arik Hadas wrote:
> >>>
> >>> Adam,
> >>
> >>
> >> :)  You seem upset.  Sorry if I touched on a nerve...
> >>
> >>> Just out of curiosity: when you write "v2v has promised" - what
exactly
> >>> do you
> >>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar
and I
> >>> that
> >>> implemented the integration with virt-v2v? I'm not aware of such a
> >>> promise by
> >>> any of these options :)
> >>
> >>
> >> Some history...
> >>
> >> Earlier this year Nir, Francesco (added), Shahar, and I began
> >> discussing the similarities between what storage needed to do with
> >> external commands and what was designed specifically for v2v.  I am
> >> not sure if you were involved in the project at that time.  The plan
> >> was to create common infrastructure that could be extended to fit the
> >> unique needs of the verticals.  The v2v code was going to be moved
> >> over to the new infrastructure (see [1]) and the only thing that
> >> stopped the initial patch was lack of a VMWare testing environment for
> >> verification.
> >>
> >> At that time storage refocused on developing verbs that used the new
> >> infrastructure and have been maintaining its suitability for general
> >> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
> >> item and much more difficult now due to the early missed opportunity.
> >>
> >>> Anyway, let's say that you were given such a promise by someone and
thus
> >>> consider that mechanism to be deprecated - it doesn't really matter.
> >>
> >>
> >> I may be biased but I think my opinion does matter.
> >>
> >>> The current implementation doesn't well fit to this flow (it requires
> >>> per-volume job, it creates leases that are not needed for template's
> >>> disks,
> >>> ...) and with the "next-gen API" with proper support for virt flows
not
> >>> even
> >>> being discussed with us (and iiuc also not with the infra team) yet, I
> >>> don't
> >>> understand what do you suggest except for some strong, though
irrelevant,
> >>> statements.
> >>
> >>
> >> If you are willing to engage in a good-faith technical discussion I am
> >> sure I can help you to understand.  These operations to storage demand
> >> some form of locking protection.  If volume leases aren't appropriate
then
> >> perhaps we should use the VM Leases / xleases that Nir is finishing
> >> off for 4.1 now.
> >>
> >>> I suggest loud and clear to reuse (not to add dependencies, not to
> >>> enhance, ..)
> >>> an existing mechanism for a very similar flow of virt-v2v that works
well
> >>> and
> >>> simple.
> >>
> >>
> >> I clearly remember discussions involving infra (hello Oved), virt
> >> (hola Michal), and storage where we decided that new APIs performing
> >> async operations involving external commands should use the HostJobs
> >> infrastructure instead of adding more information to Host Stats.
> >> These were the "famous" entity polling meetings.
>
> We discussed these issues behind close doors, not in the public mailing
list,
> so it is not surprising that people do not know about the agreements we
had.
>

The core team was there. So it is surprising.

> >>
> >> Of course plans can change but I have never been looped into any such
> >> discussions.
> >>
> >
> > Well, I think that when someone builds a good infrastructure he first
needs
> > to talk to all consumers and make sure it fits.
> > In this case it seems like most work was done to fit the storage
use-case,
> > and now you check whether it can fit others as well
>
> The jobs framework is generic and can be used for any subsystem,
> there is nothing related to storage about it. But modifying disks *is*
> a storage operation, even if someone from the virt team worked on it.
>
> V2v is also storage operation - if we compare it with copying disks:
>
> - we create a new volume that nobody is using yet
> - if the operation fails, the disk must be in illegal state
> - if the operation fails we delete the disks
> - if the operation succeeds the volume must be legal
> - we need to limit the number of operations on a host
> - we need to detect the job state if the host becomes non-responsive
> - we may want to fence the job if the host becomes non-responsive
>   in volume jobs, we can increment the volume generation and run
>   the same job on another host.
> - we want to take a lease on storage to ensure that other hosts cannot
>   access the same entity, or that the job will fail if someone else is
using
>   this entity
> - we want to take a lease on storage, ensuring that a job cannot get
>   stuck for long time - sanlock kill the owner of a lease when storage
>   becomes inaccessible.
> - we want to report progress
>
> sysprep is less risky because the operation is faster, but on 

Re: [ovirt-devel] ovirt-node_ovirt-3.6_create-iso-el7_merged job failed

[Eyal Edri]

looks like it is trying to bring qemu 2.6 which is only available in centos
7.3 which was not released yet.
 Adding node team.

Fabian,  does node already requires rpms from 7.3?


On Dec 7, 2016 6:46 PM, "Shlomo Ben David"  wrote:

> Hi,
>
> The [1] job failed with the following error:
>
> *10:21:33 Package qemu-kvm-tools is obsoleted by qemu-kvm-tools-ev, trying to 
> install 10:qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64 instead
> 10:21:48 Error creating Live CD : Failed to build transaction : 
> 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires usbredir >= 0.7.1
> 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires seavgabios-bin >= 
> 1.9.1-4
> 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires ipxe-roms-qemu >= 
> 20160127-4
> 10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires libusbx >= 1.0.19
> 10:21:48 ERROR: ISO build failed.*
>
>
> [1] - http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_
> create-iso-el7_merged/171/console
>
> Best Regards,
>
> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
> RHCSA | RHCE
> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>
> OPEN SOURCE - 1 4 011 && 011 4 1
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] ovirt-node_ovirt-3.6_create-iso-el7_merged job failed

[Shlomo Ben David]

Hi,

The [1] job failed with the following error:

*10:21:33 Package qemu-kvm-tools is obsoleted by qemu-kvm-tools-ev,
trying to install 10:qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64 instead
10:21:48 Error creating Live CD : Failed to build transaction :
10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires usbredir >= 0.7.1
10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires seavgabios-bin >= 1.9.1-4
10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires ipxe-roms-qemu
>= 20160127-4
10:21:48 10:qemu-kvm-ev-2.6.0-27.1.el7.x86_64 requires libusbx >= 1.0.19
10:21:48 ERROR: ISO build failed.*


[1] -
http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/171/console

Best Regards,

Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
RHCSA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)

OPEN SOURCE - 1 4 011 && 011 4 1
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Vdsm tests are 4X times faster on travis

[Barak Korren]

On 7 December 2016 at 14:15, Nir Soffer  wrote:
>
> Last week it was 19-20 mintues, great improvement!
>
there are a couple of other things we might try soon, that will,
perhaps, help us shave off another 2-3 minutes...




-- 
Barak Korren
bkor...@redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Vdsm tests are 4X times faster on travis

[Barak Korren]

On 7 December 2016 at 14:15, Nir Soffer  wrote:
> On Mon, Dec 5, 2016 at 10:11 AM, Barak Korren  wrote:
>> On 5 December 2016 at 10:07, Nir Soffer  wrote:
>>>
>>> 20 seconds setup sounds great.
>>>
>>> Can we try the patches with vdsm builds?
>>
>> We'll probably merge today, if not, I'll manually cherry-pick this for
>> the vdsm jobs.
>
> With all patches merged, builds take now:
>
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/4373/console
> 00:07:13.065 Finished: SUCCESS
>
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5991/console
> 00:08:19.035 Finished: SUCCESS
>
> Last week it was 19-20 mintues, great improvement!



>
> What is missing now small improvement in the logs - add log for each part of 
> the
> build with the time it took.
>
> - setup
> - run script
> - cleanup
>
> The run script part is something that only project maintainers can optimize, 
> the
> rest can be optimized only by ci maintainers.

WRT to times - mock_runner.sh does print those out to the output page
(surrounded by many asterisks and other symbols...)

WRT separation of logs - we already mostly have that, you can see
individual step logs in the job artifacts. For example for one of the
jobs above:

http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5991/artifact/exported-artifacts/logs/mocker-fedora-24-x86_64.fc24.check-patch.sh/check-patch.sh.log

But yeah, we could probably make the output in the main Jenkins output
page better. That would require potentially breaking changes to
"mock_runner.sh", so I'd rather focus on ultimately replacing it...

We do already have https://ovirt-jira.atlassian.net/browse/OVIRT-682
for discussing this issue.

> I think we should have metrics collection system and keep these times
> so we can detect regressions and improvement easily. But the first step
> is measuring the time.

Jenkins (partially) gives us that, see here:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/buildTimeTrend

I completely agree that the UX here is not as good as it can and
should be, and we do have plans to make it A LOT better, please bare
with us in the meantime...

-- 
Barak Korren
bkor...@redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Barak Korren]

On 7 December 2016 at 16:59, Yedidyah Bar David  wrote:
>
> Perhaps we can have (instead? in addition?) a button on gerrit
> "trigger build-artifacts" that will run the standard-CI build-artifacts?
> Or some other new script we'll add to standard-CI? Because current
> build-artifacts is expected to be ran only after merge, and the
> publisher reads it. Perhaps e.g. "build-scratch-artifacts".

Strictly speaking, this is possible.
Practically - I do not know of a way to do this besides starting to
maintain a custom Gerrit plug-in, which is something I'm not eager to
try... (An I think neither is anyone else from the infra team...)

But go ahead and create a Jira ticket, why not...

-- 
Barak Korren
bkor...@redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Eyal Edri]

On Wed, Dec 7, 2016 at 4:59 PM, Yedidyah Bar David  wrote:

> On Wed, Dec 7, 2016 at 4:39 PM, Eyal Edri  wrote:
> > FYI,
> >
> > After hearing that this might be useful to many developers, I wanted to
> > bring to your attention the ovirt-engine master build job from patch [1]
> > which allows you to build new rpms from an open ovirt-engine patch on
> > Gerrit.
> >
> > Its was created as temp job for a team who needed it a few months ago,
> but
> > seems it might useful for other teams as well, so we decided to publish
> it
> > even though its not standardized yet as part of our standard CI system.
> >
> > I hope you can find it useful for your teams.
>
> Never heard about it, sounds useful.
>
> We have too many jobs (for me) to know them all.
>
> So I use 'search'. What to search for? Usually the project name, but some
> projects have many jobs too.
>

There are no more special hidden jobs :) If I'll find out on something
useful i'll publish it.


>
> There are jobs named something_manual, 87 currently. Some (many? all?
> those I know) get a tarball as input. I don't like this personally,
> as I can't know, when using a result of such a job, what was in the
> tarball. Much better to have to push first to gerrit and supply
> that (or commit hash, or something like that).
>
> Some of them are called something_any_something_manual. Are these
> special? What's 'any'?
>

These jobs replaced the old 'manual' jobs for building official rpms, not
all maintainers use them.
see [1]

[1]
http://infra-docs.readthedocs.io/en/latest/CI/Build_and_test_standards.html#build-artifacts-manual-sh


>
> Perhaps we can have (instead? in addition?) a button on gerrit
> "trigger build-artifacts" that will run the standard-CI build-artifacts?
> Or some other new script we'll add to standard-CI? Because current
> build-artifacts is expected to be ran only after merge, and the
> publisher reads it. Perhaps e.g. "build-scratch-artifacts".
>

Not sure Gerrit supports it, but feel free to open a ticket with ideas and
we'll see if there are options to add it.
I'm not sure how often people need artifacts during devel, but you can
easily add it to check-patch.sh script
and the jenkins job will archive the rpms for you.


>
> >
> >
> > [1]
> > http://jenkins.ovirt.org/job/ovirt-engine_master_build-
> artifacts-el7-x86_64_build_from_patch/
> >
> > --
> > Eyal Edri
> > Associate Manager
> > RHV DevOps
> > EMEA ENG Virtualization R
> > Red Hat Israel
> >
> > phone: +972-9-7692018
> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
> --
> Didi
>



-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Yedidyah Bar David]

On Wed, Dec 7, 2016 at 4:39 PM, Eyal Edri  wrote:
> FYI,
>
> After hearing that this might be useful to many developers, I wanted to
> bring to your attention the ovirt-engine master build job from patch [1]
> which allows you to build new rpms from an open ovirt-engine patch on
> Gerrit.
>
> Its was created as temp job for a team who needed it a few months ago, but
> seems it might useful for other teams as well, so we decided to publish it
> even though its not standardized yet as part of our standard CI system.
>
> I hope you can find it useful for your teams.

Never heard about it, sounds useful.

We have too many jobs (for me) to know them all.

So I use 'search'. What to search for? Usually the project name, but some
projects have many jobs too.

There are jobs named something_manual, 87 currently. Some (many? all?
those I know) get a tarball as input. I don't like this personally,
as I can't know, when using a result of such a job, what was in the
tarball. Much better to have to push first to gerrit and supply
that (or commit hash, or something like that).

Some of them are called something_any_something_manual. Are these
special? What's 'any'?

Perhaps we can have (instead? in addition?) a button on gerrit
"trigger build-artifacts" that will run the standard-CI build-artifacts?
Or some other new script we'll add to standard-CI? Because current
build-artifacts is expected to be ran only after merge, and the
publisher reads it. Perhaps e.g. "build-scratch-artifacts".

>
>
> [1]
> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64_build_from_patch/
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel



-- 
Didi
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The engine build job you probably didn't know exists

[Roy Golan]

On 7 December 2016 at 16:39, Eyal Edri  wrote:

> FYI,
>
> After hearing that this might be useful to many developers, I wanted to
> bring to your attention the ovirt-engine master build job from patch [1]
> which allows you to build new rpms from an open ovirt-engine patch on
> Gerrit.
>
> Its was created as temp job for a team who needed it a few months ago, but
> seems it might useful for other teams as well, so we decided to publish it
> even though its not standardized yet as part of our standard CI system.
>
> I hope you can find it useful for your teams.
>
>
> [1] http://jenkins.ovirt.org/job/ovirt-engine_master_build-
> artifacts-el7-x86_64_build_from_patch/
>

+1

It will be also very useful if the job can create a tar.gz instead of rpm
with all the artifacts so we can untar it to our development PREFIX.

>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] The engine build job you probably didn't know exists

[Eyal Edri]

FYI,

After hearing that this might be useful to many developers, I wanted to
bring to your attention the ovirt-engine master build job from patch [1]
which allows you to build new rpms from an open ovirt-engine patch on
Gerrit.

Its was created as temp job for a team who needed it a few months ago, but
seems it might useful for other teams as well, so we decided to publish it
even though its not standardized yet as part of our standard CI system.

I hope you can find it useful for your teams.


[1]
http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_64_build_from_patch/


-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM review & development tools

[Martin Sivak]

Hi,

> 1) gerrit (stay as we are),

Gerrit with improved merging procedures will be a very good tool. It
preserves history of reviews and allows me to display differences
between patchsets easily (and lets not forget topic branches across
projects).

> 2) github & reviewable,

Probably ok as well, I have never used reviewable. I can currently
compare and correlate patches faster using gerrit though. Also lets
not forget about the ecosystem, having the same gerrit instance for
all projects makes it possible to use topic branches for related
patches.

> 3) mailing list.

Been there, done that, ran away. Mailing list makes the management
hard. Especially with patch revisions and large patch sets. There is
no common picture (everybody has to correlate comments on their own),
no unified history or changeset differences and no per line comments
in the current state of things. Also many people use emails providers
that do not do a good job in preserving complicated thread
hierarchies.

We used to use mailing list in the Anaconda project and moved to GitHub.

TLDR: pure mails are hard unless you have a tree of maintainers to
filter it for you and lots of time to track information down

PS:

> c) freedom - It's up to us to decide how would we handle such
> development.

Treating VDSM as a completely separate project would be a big change
to the paradigm, it is currently treated as a "dumb" host proxy (just
a provider of low level commands with logic in the engine) by many
teams.

Martin

On Wed, Dec 7, 2016 at 1:06 PM, Martin Polednik  wrote:
> Hello developers,
>
> this e-mail is mostly aimed at VDSM contributors (current and
> potential) and should serve as continuation of the idea started at our
> weekly meeting.
>
> We've had a proposal of moving the project to github[1] and using
> reviewable[2] as a code review due to github's code review interface.
>
> I would like to start a *discussion* regarding how we would like to
> develop VDSM in future. So far, the suggested options were (1st
> implied):
>
> 1) gerrit (stay as we are),
> 2) github & reviewable,
> 3) mailing list.
>
> What would be your favorite review & development tool and why? Do you
> hate any of them? Let the flamewar^Wconstructive discussion begin! :)
>
>
> 
> My preferred tool is mailing list with the main tree mirrored to
> github. Why mailing list in 2016 (almost 2017)?
>
> a) stack - We're built on top of libvirt, libvirt is
> built on top of qemu and qemu utilizes kvm which is a kernel module.
> Each of this projects uses mailing lists for development.
>
> b) tooling - Everyone is free to use tools of choice. Any sane e-mail
> client can handle mailing list patches, and it's up to reviewers to
> choose the best way to handle the review. As for sending the patches,
> there is the universal fallback in the form of git-send-email.
>
> c) freedom - It's up to us to decide how would we handle such
> development. As many system-level projects already use mailing lists
> (see a), there is enough inspiration for workflow design[3].
>
> d) accessibility - VDSM is in unfortunate position between "cool",
> high level projects and "boring" low level projects. I believe that we
> should be more accessible to the developers from below the stack
> rather than to general public. Having unified workflow that doesn't
> require additional accounts and is compatible with their workflows
> makes that easier.
> 
>
>
> [1] https://github.com/
> [2] https://reviewable.io/
> [3] e.g. https://www.kernel.org/doc/Documentation/SubmittingPatches
>
> Regards,
> mpolednik
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] ovirt_4.0_he-system-tests job failed

[Gil Shinar]

rerun the 4.0 nightly publisher and then rerun this job and now looks like
it has past the installation phase.

On Wed, Dec 7, 2016 at 2:40 PM, Eyal Edri  wrote:

> Gil - can you check if the 4.0 publishers jobs to snapshot are fixed to
> deploy the old pkg?
>
> On Wed, Dec 7, 2016 at 1:53 PM, Shlomo Ben David 
> wrote:
>
>> Hi,
>>
>> The [1] job failed with the following error:
>>
>> 2016-12-07 02:40:43,805::ssh.py::ssh::96::lago.ssh::DEBUG::Command 8b25fff2 
>> on lago-he-basic-suite-4-0-host0  errors:
>>  Error: Package: 
>> ovirt-hosted-engine-setup-2.0.4.2-0.0.master.20161201083322.gita375924.el7.centos.noarch
>>  (alocalsync)
>>Requires: ovirt-engine-sdk-python >= 3.6.3.0
>>
>>
>> [1] - http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/584/console
>>
>>
>> Best Regards,
>>
>> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
>> RHCSA | RHCE
>> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>>
>> OPEN SOURCE - 1 4 011 && 011 4 1
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Nir Soffer]

On Wed, Dec 7, 2016 at 10:17 AM, Oved Ourfali  wrote:
>
>
> On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:
>>
>> On 06/12/16 22:06 +0200, Arik Hadas wrote:
>>>
>>> Adam,
>>
>>
>> :)  You seem upset.  Sorry if I touched on a nerve...
>>
>>> Just out of curiosity: when you write "v2v has promised" - what exactly
>>> do you
>>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I
>>> that
>>> implemented the integration with virt-v2v? I'm not aware of such a
>>> promise by
>>> any of these options :)
>>
>>
>> Some history...
>>
>> Earlier this year Nir, Francesco (added), Shahar, and I began
>> discussing the similarities between what storage needed to do with
>> external commands and what was designed specifically for v2v.  I am
>> not sure if you were involved in the project at that time.  The plan
>> was to create common infrastructure that could be extended to fit the
>> unique needs of the verticals.  The v2v code was going to be moved
>> over to the new infrastructure (see [1]) and the only thing that
>> stopped the initial patch was lack of a VMWare testing environment for
>> verification.
>>
>> At that time storage refocused on developing verbs that used the new
>> infrastructure and have been maintaining its suitability for general
>> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
>> item and much more difficult now due to the early missed opportunity.
>>
>>> Anyway, let's say that you were given such a promise by someone and thus
>>> consider that mechanism to be deprecated - it doesn't really matter.
>>
>>
>> I may be biased but I think my opinion does matter.
>>
>>> The current implementation doesn't well fit to this flow (it requires
>>> per-volume job, it creates leases that are not needed for template's
>>> disks,
>>> ...) and with the "next-gen API" with proper support for virt flows not
>>> even
>>> being discussed with us (and iiuc also not with the infra team) yet, I
>>> don't
>>> understand what do you suggest except for some strong, though irrelevant,
>>> statements.
>>
>>
>> If you are willing to engage in a good-faith technical discussion I am
>> sure I can help you to understand.  These operations to storage demand
>> some form of locking protection.  If volume leases aren't appropriate then
>> perhaps we should use the VM Leases / xleases that Nir is finishing
>> off for 4.1 now.
>>
>>> I suggest loud and clear to reuse (not to add dependencies, not to
>>> enhance, ..)
>>> an existing mechanism for a very similar flow of virt-v2v that works well
>>> and
>>> simple.
>>
>>
>> I clearly remember discussions involving infra (hello Oved), virt
>> (hola Michal), and storage where we decided that new APIs performing
>> async operations involving external commands should use the HostJobs
>> infrastructure instead of adding more information to Host Stats.
>> These were the "famous" entity polling meetings.

We discussed these issues behind close doors, not in the public mailing list,
so it is not surprising that people do not know about the agreements we had.

>>
>> Of course plans can change but I have never been looped into any such
>> discussions.
>>
>
> Well, I think that when someone builds a good infrastructure he first needs
> to talk to all consumers and make sure it fits.
> In this case it seems like most work was done to fit the storage use-case,
> and now you check whether it can fit others as well

The jobs framework is generic and can be used for any subsystem,
there is nothing related to storage about it. But modifying disks *is*
a storage operation, even if someone from the virt team worked on it.

V2v is also storage operation - if we compare it with copying disks:

- we create a new volume that nobody is using yet
- if the operation fails, the disk must be in illegal state
- if the operation fails we delete the disks
- if the operation succeeds the volume must be legal
- we need to limit the number of operations on a host
- we need to detect the job state if the host becomes non-responsive
- we may want to fence the job if the host becomes non-responsive
  in volume jobs, we can increment the volume generation and run
  the same job on another host.
- we want to take a lease on storage to ensure that other hosts cannot
  access the same entity, or that the job will fail if someone else is using
  this entity
- we want to take a lease on storage, ensuring that a job cannot get
  stuck for long time - sanlock kill the owner of a lease when storage
  becomes inaccessible.
- we want to report progress

sysprep is less risky because the operation is faster, but on storage even
fast operation can get stuck for minutes.

We need to agree on a standard way to do such operations that is safe enough
and can be managed on the engine side.

> IMO it makes much more sense to use events where possible (and you've
> promised to use those as well, but I don't see you doing that...). v2v
> should use 

Re: [ovirt-devel] VDSM review & development tools

[Barak Korren]

On 7 December 2016 at 14:06, Martin Polednik  wrote:
>
> What would be your favorite review & development tool and why? Do you
> hate any of them? Let the flamewar^Wconstructive discussion begin! :)
>

If you've read any of the recent threads I've started on devel, you
may know we (the infra team) are working on integration Zuul into the
CI system in order to provide merge gating (As well as other
benefits).

A truly efficient merge gating is the kind that you do cross-project.
This means that you setup the gate so that it looks at all the pending
changes across all the projects, builds the system as it would look
with all those changes merged, and checks it.

The benefit of this system is that you know that not only all the
project branches are stable, but also that when you bring artifacts
built from those branches together, the system as a whole works.

As for now, Zuul is the only system I know of that does good
cross-project merge gating. And, for the time being, it does not
support anything other then Gerrit. So I recommend we stay with Gerrit
for now.

-- 
Barak Korren
bkor...@redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] ovirt_4.0_he-system-tests job failed

[Eyal Edri]

Gil - can you check if the 4.0 publishers jobs to snapshot are fixed to
deploy the old pkg?

On Wed, Dec 7, 2016 at 1:53 PM, Shlomo Ben David 
wrote:

> Hi,
>
> The [1] job failed with the following error:
>
> 2016-12-07 02:40:43,805::ssh.py::ssh::96::lago.ssh::DEBUG::Command 8b25fff2 
> on lago-he-basic-suite-4-0-host0  errors:
>  Error: Package: 
> ovirt-hosted-engine-setup-2.0.4.2-0.0.master.20161201083322.gita375924.el7.centos.noarch
>  (alocalsync)
>Requires: ovirt-engine-sdk-python >= 3.6.3.0
>
>
> [1] - http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/584/console
>
>
> Best Regards,
>
> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
> RHCSA | RHCE
> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>
> OPEN SOURCE - 1 4 011 && 011 4 1
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>



-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] test-repo_ovirt_experimental_4.0

[Gil Shinar]

Looks like it is from day 1
I'll try to see why its like that. Maybe someone has forgotten to copy the
artifacts into that folder

On Wed, Dec 7, 2016 at 2:09 PM, Eyal Edri  wrote:

>
>
> On Wed, Dec 7, 2016 at 1:51 PM, Gil Shinar  wrote:
>
>> Sorry for opening a different mailing thread with Juan without CCIng you
>> as well.
>> I think we have an issue with the ovirt-engine-sdk job for 4.0. As you
>> can see in [1], the RPMs this job is generating has different names than
>> what master job [2] is generating.
>>
>> Having said the above, Now [3] I see that it has the right name.
>>
>
> Yes, I guess last time it run was before we split the jobs.
> What about the warn in deploy to experimental? is it harmless? since i now
> see job back to stable. (the test experimental)
>
>
>>
>> [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-
>> artifacts-el7-x86_64/72/
>> [2] http://jenkins.ovirt.org/job/ovirt-engine-sdk_master_bui
>> ld-artifacts-el7-x86_64/
>> [3] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-
>> artifacts-el7-x86_64/
>>
>> On Wed, Dec 7, 2016 at 1:03 PM, Eyal Edri  wrote:
>>
>>> There might be a bug in the sdk 4.0 job, the experimental flow didn't
>>> publish artifacts [1] since it couldn't find them in exported-artifacts:
>>>
>>>
>>> WARN: No artifacts found that match the file pattern
>>> "exported-artifacts/**". Configuration error?
>>> WARN: ?exported-artifacts/**? doesn?t match anything, but ?**? does.
>>> Perhaps that?s what you mean?
>>>
>>> [1] http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_4.
>>> 0/5281/console
>>>
>>> On Wed, Dec 7, 2016 at 12:56 PM, Eyal Edri  wrote:
>>>
 I triggered the sdk job [1] so it should publish the rpms to 4.0 repo
 soon,
 Lets give it one hour and see if it fixed the issue.

 Regadless, please open a ticket in JIRA to investigate why the rpm was
 deleted.

 [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-
 artifacts-el7-x86_64/73/console

 On Wed, Dec 7, 2016 at 12:46 PM, Eyal Edri  wrote:

> try triggering the ovirt sdk job build artifacts for 4.0.
> maybe for some reason it was removed from experimental,  we will need
> to check what caused it to be removed.
>
> On Dec 7, 2016 12:39, "Shlomo Ben David"  wrote:
>
>> Hi All,
>>
>> The [1] job is failing with the following error:
>>
>> *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine 
>> ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*'
>> Error: Package: 
>> ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch 
>> (alocalsync)
>>Requires: ovirt-engine-sdk-python >= 3.6.0.0
>>  [0m
>>  [36m* [Thread-4] Deploy VM lago-basic-suite-4-0-engine:  [31mERROR 
>> [0m (in 0:01:51) [0m
>>  [31mError while running thread
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
>> _ret_via_queue
>> queue.put({'return': func()})
>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
>> _deploy_host
>> host.name (),
>> RuntimeError: 
>> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>>  [36m  # Deploy environment:  [31mERROR [0m (in 0:01:51) [0m
>>  [36m@ Deploy oVirt environment:  [31mERROR [0m (in 0:01:51) [0m
>>  [31mError occured, aborting
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in 
>> do_run
>> self.cli_plugins[args.ovirtverb].do_run(args)
>>   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, 
>> in do_run
>> self._do_run(**vars(args))
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in 
>> wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in 
>> wrapper
>> return func(*args, prefix=prefix, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in 
>> do_deploy
>> prefix.deploy()
>>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, 
>> in wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 
>> 67, in wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 
>> 198, in deploy
>> return super(OvirtPrefix, 

Re: [ovirt-devel] Vdsm tests are 4X times faster on travis

[Nir Soffer]

On Mon, Dec 5, 2016 at 10:11 AM, Barak Korren  wrote:
> On 5 December 2016 at 10:07, Nir Soffer  wrote:
>>
>> 20 seconds setup sounds great.
>>
>> Can we try the patches with vdsm builds?
>
> We'll probably merge today, if not, I'll manually cherry-pick this for
> the vdsm jobs.

With all patches merged, builds take now:

http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/4373/console
00:07:13.065 Finished: SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5991/console
00:08:19.035 Finished: SUCCESS

Last week it was 19-20 mintues, great improvement!

What is missing now small improvement in the logs - add log for each part of the
build with the time it took.

- setup
- run script
- cleanup

The run script part is something that only project maintainers can optimize, the
rest can be optimized only by ci maintainers.

I think we should have metrics collection system and keep these times
so we can detect regressions and improvement easily. But the first step
is measuring the time.

Nir
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] test-repo_ovirt_experimental_4.0

[Eyal Edri]

On Wed, Dec 7, 2016 at 1:51 PM, Gil Shinar  wrote:

> Sorry for opening a different mailing thread with Juan without CCIng you
> as well.
> I think we have an issue with the ovirt-engine-sdk job for 4.0. As you can
> see in [1], the RPMs this job is generating has different names than what
> master job [2] is generating.
>
> Having said the above, Now [3] I see that it has the right name.
>

Yes, I guess last time it run was before we split the jobs.
What about the warn in deploy to experimental? is it harmless? since i now
see job back to stable. (the test experimental)


>
> [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_
> build-artifacts-el7-x86_64/72/
> [2] http://jenkins.ovirt.org/job/ovirt-engine-sdk_master_
> build-artifacts-el7-x86_64/
> [3] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_
> build-artifacts-el7-x86_64/
>
> On Wed, Dec 7, 2016 at 1:03 PM, Eyal Edri  wrote:
>
>> There might be a bug in the sdk 4.0 job, the experimental flow didn't
>> publish artifacts [1] since it couldn't find them in exported-artifacts:
>>
>>
>> WARN: No artifacts found that match the file pattern
>> "exported-artifacts/**". Configuration error?
>> WARN: ?exported-artifacts/**? doesn?t match anything, but ?**? does.
>> Perhaps that?s what you mean?
>>
>> [1] http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_4.
>> 0/5281/console
>>
>> On Wed, Dec 7, 2016 at 12:56 PM, Eyal Edri  wrote:
>>
>>> I triggered the sdk job [1] so it should publish the rpms to 4.0 repo
>>> soon,
>>> Lets give it one hour and see if it fixed the issue.
>>>
>>> Regadless, please open a ticket in JIRA to investigate why the rpm was
>>> deleted.
>>>
>>> [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-
>>> artifacts-el7-x86_64/73/console
>>>
>>> On Wed, Dec 7, 2016 at 12:46 PM, Eyal Edri  wrote:
>>>
 try triggering the ovirt sdk job build artifacts for 4.0.
 maybe for some reason it was removed from experimental,  we will need
 to check what caused it to be removed.

 On Dec 7, 2016 12:39, "Shlomo Ben David"  wrote:

> Hi All,
>
> The [1] job is failing with the following error:
>
> *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine 
> ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*'
> Error: Package: 
> ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch 
> (alocalsync)
>Requires: ovirt-engine-sdk-python >= 3.6.0.0
>  [0m
>  [36m* [Thread-4] Deploy VM lago-basic-suite-4-0-engine:  [31mERROR 
> [0m (in 0:01:51) [0m
>  [31mError while running thread
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
> _ret_via_queue
> queue.put({'return': func()})
>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
> _deploy_host
> host.name (),
> RuntimeError: 
> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>  [36m  # Deploy environment:  [31mERROR [0m (in 0:01:51) [0m
>  [36m@ Deploy oVirt environment:  [31mERROR [0m (in 0:01:51) [0m
>  [31mError occured, aborting
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in 
> do_run
> self.cli_plugins[args.ovirtverb].do_run(args)
>   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, 
> in do_run
> self._do_run(**vars(args))
>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in 
> wrapper
> return func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in 
> wrapper
> return func(*args, prefix=prefix, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in 
> do_deploy
> prefix.deploy()
>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
> wrapper
> return func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 
> 67, in wrapper
> return func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 
> 198, in deploy
> return super(OvirtPrefix, self).deploy()
>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
> wrapper
> return func(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in 
> deploy
> self._deploy_host, self.virt_env.get_vms().values()
>   File 

[ovirt-devel] VDSM review & development tools

[Martin Polednik]

Hello developers,

this e-mail is mostly aimed at VDSM contributors (current and
potential) and should serve as continuation of the idea started at our
weekly meeting.

We've had a proposal of moving the project to github[1] and using
reviewable[2] as a code review due to github's code review interface.

I would like to start a *discussion* regarding how we would like to
develop VDSM in future. So far, the suggested options were (1st
implied):

1) gerrit (stay as we are),
2) github & reviewable,
3) mailing list.

What would be your favorite review & development tool and why? Do you
hate any of them? Let the flamewar^Wconstructive discussion begin! :)



My preferred tool is mailing list with the main tree mirrored to
github. Why mailing list in 2016 (almost 2017)?

a) stack - We're built on top of libvirt, libvirt is
built on top of qemu and qemu utilizes kvm which is a kernel module.
Each of this projects uses mailing lists for development.

b) tooling - Everyone is free to use tools of choice. Any sane e-mail
client can handle mailing list patches, and it's up to reviewers to
choose the best way to handle the review. As for sending the patches,
there is the universal fallback in the form of git-send-email.

c) freedom - It's up to us to decide how would we handle such
development. As many system-level projects already use mailing lists
(see a), there is enough inspiration for workflow design[3].

d) accessibility - VDSM is in unfortunate position between "cool",
high level projects and "boring" low level projects. I believe that we
should be more accessible to the developers from below the stack
rather than to general public. Having unified workflow that doesn't
require additional accounts and is compatible with their workflows
makes that easier.



[1] https://github.com/
[2] https://reviewable.io/
[3] e.g. https://www.kernel.org/doc/Documentation/SubmittingPatches

Regards,
mpolednik
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] New test failure on travis

[Nir Soffer]

Looks like we need @brokentest("reason...", name="TRAVIC_CI") on this:

See https://travis-ci.org/oVirt/vdsm/jobs/181933329

==

ERROR: test suite for 

--

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run

self.setUp()

  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp

self.setupContext(ancestor)

  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in
setupContext

try_run(context, names)

  File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run

return func()

  File "/vdsm/tests/testValidation.py", line 191, in wrapper

return f(*args, **kwargs)

  File "/vdsm/tests/testValidation.py", line 97, in wrapper

return f(*args, **kwargs)

  File "/vdsm/tests/network/nmdbus_test.py", line 48, in setup_module

NMDbus.init()

  File "/vdsm/lib/vdsm/network/nm/nmdbus/__init__.py", line 33, in init

NMDbus.bus = dbus.SystemBus()

  File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 194, in __new__

private=private)

  File "/usr/lib64/python2.7/site-packages/dbus/_dbus.py", line 100, in __new__

bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)

  File "/usr/lib64/python2.7/site-packages/dbus/bus.py", line 122, in __new__

bus = cls._new_for_bus(address_or_type, mainloop=mainloop)

DBusException: org.freedesktop.DBus.Error.FileNotFound: Failed to
connect to socket /var/run/dbus/system_bus_socket: No such file or
directory

 >> begin captured logging << 

2016-12-07 11:48:33,458 DEBUG (MainThread) [root] /usr/bin/taskset
--cpu-list 0-1 /bin/systemctl status NetworkManager (cwd None)
(commands:69)

2016-12-07 11:48:33,465 DEBUG (MainThread) [root] FAILED:  =
'Failed to get D-Bus connection: Operation not permitted\n';  = 1
(commands:93)

2016-12-07 11:48:33,465 DEBUG (MainThread) [root] /usr/bin/taskset
--cpu-list 0-1 /bin/systemctl start NetworkManager (cwd None)
(commands:69)

2016-12-07 11:48:33,470 DEBUG (MainThread) [root] FAILED:  =
'Failed to get D-Bus connection: Operation not permitted\n';  = 1
(commands:93)

- >> end captured logging << -
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] ovirt_4.0_he-system-tests job failed

[Shlomo Ben David]

Hi,

The [1] job failed with the following error:

2016-12-07 02:40:43,805::ssh.py::ssh::96::lago.ssh::DEBUG::Command
8b25fff2 on lago-he-basic-suite-4-0-host0  errors:
 Error: Package:
ovirt-hosted-engine-setup-2.0.4.2-0.0.master.20161201083322.gita375924.el7.centos.noarch
(alocalsync)
   Requires: ovirt-engine-sdk-python >= 3.6.3.0


[1] - http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/584/console


Best Regards,

Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
RHCSA | RHCE
IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)

OPEN SOURCE - 1 4 011 && 011 4 1
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] test-repo_ovirt_experimental_4.0

[Gil Shinar]

Sorry for opening a different mailing thread with Juan without CCIng you as
well.
I think we have an issue with the ovirt-engine-sdk job for 4.0. As you can
see in [1], the RPMs this job is generating has different names than what
master job [2] is generating.

Having said the above, Now [3] I see that it has the right name.

[1]
http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-artifacts-el7-x86_64/72/
[2]
http://jenkins.ovirt.org/job/ovirt-engine-sdk_master_build-artifacts-el7-x86_64/
[3]
http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-artifacts-el7-x86_64/

On Wed, Dec 7, 2016 at 1:03 PM, Eyal Edri  wrote:

> There might be a bug in the sdk 4.0 job, the experimental flow didn't
> publish artifacts [1] since it couldn't find them in exported-artifacts:
>
>
> WARN: No artifacts found that match the file pattern
> "exported-artifacts/**". Configuration error?
> WARN: ?exported-artifacts/**? doesn?t match anything, but ?**? does.
> Perhaps that?s what you mean?
>
> [1] http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_
> 4.0/5281/console
>
> On Wed, Dec 7, 2016 at 12:56 PM, Eyal Edri  wrote:
>
>> I triggered the sdk job [1] so it should publish the rpms to 4.0 repo
>> soon,
>> Lets give it one hour and see if it fixed the issue.
>>
>> Regadless, please open a ticket in JIRA to investigate why the rpm was
>> deleted.
>>
>> [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-
>> artifacts-el7-x86_64/73/console
>>
>> On Wed, Dec 7, 2016 at 12:46 PM, Eyal Edri  wrote:
>>
>>> try triggering the ovirt sdk job build artifacts for 4.0.
>>> maybe for some reason it was removed from experimental,  we will need to
>>> check what caused it to be removed.
>>>
>>> On Dec 7, 2016 12:39, "Shlomo Ben David"  wrote:
>>>
 Hi All,

 The [1] job is failing with the following error:

 *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine 
 ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*'
 Error: Package: 
 ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch 
 (alocalsync)
Requires: ovirt-engine-sdk-python >= 3.6.0.0
  [0m
  [36m* [Thread-4] Deploy VM lago-basic-suite-4-0-engine:  [31mERROR 
 [0m (in 0:01:51) [0m
  [31mError while running thread
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
 _ret_via_queue
 queue.put({'return': func()})
   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
 _deploy_host
 host.name (),
 RuntimeError: 
 /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
  failed with status 1 on lago-basic-suite-4-0-engine [0m
  [36m  # Deploy environment:  [31mERROR [0m (in 0:01:51) [0m
  [36m@ Deploy oVirt environment:  [31mERROR [0m (in 0:01:51) [0m
  [31mError occured, aborting
 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in 
 do_run
 self.cli_plugins[args.ovirtverb].do_run(args)
   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, 
 in do_run
 self._do_run(**vars(args))
   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in 
 wrapper
 return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in 
 wrapper
 return func(*args, prefix=prefix, **kwargs)
   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in 
 do_deploy
 prefix.deploy()
   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
 wrapper
 return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 67, 
 in wrapper
 return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, 
 in deploy
 return super(OvirtPrefix, self).deploy()
   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
 wrapper
 return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in 
 deploy
 self._deploy_host, self.virt_env.get_vms().values()
   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in 
 invoke_in_parallel
 vt.join_all()
   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
 _ret_via_queue
 queue.put({'return': func()})
   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
 _deploy_host
 host.name (),
 

Re: [ovirt-devel] test-repo_ovirt_experimental_4.0

[Eyal Edri]

There might be a bug in the sdk 4.0 job, the experimental flow didn't
publish artifacts [1] since it couldn't find them in exported-artifacts:


WARN: No artifacts found that match the file pattern
"exported-artifacts/**". Configuration error?
WARN: ?exported-artifacts/**? doesn?t match anything, but ?**? does.
Perhaps that?s what you mean?

[1]
http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_4.0/5281/console

On Wed, Dec 7, 2016 at 12:56 PM, Eyal Edri  wrote:

> I triggered the sdk job [1] so it should publish the rpms to 4.0 repo soon,
> Lets give it one hour and see if it fixed the issue.
>
> Regadless, please open a ticket in JIRA to investigate why the rpm was
> deleted.
>
> [1] http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_
> build-artifacts-el7-x86_64/73/console
>
> On Wed, Dec 7, 2016 at 12:46 PM, Eyal Edri  wrote:
>
>> try triggering the ovirt sdk job build artifacts for 4.0.
>> maybe for some reason it was removed from experimental,  we will need to
>> check what caused it to be removed.
>>
>> On Dec 7, 2016 12:39, "Shlomo Ben David"  wrote:
>>
>>> Hi All,
>>>
>>> The [1] job is failing with the following error:
>>>
>>> *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine 
>>> ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*'
>>> Error: Package: 
>>> ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch 
>>> (alocalsync)
>>>Requires: ovirt-engine-sdk-python >= 3.6.0.0
>>>  [0m
>>>  [36m* [Thread-4] Deploy VM lago-basic-suite-4-0-engine:  [31mERROR [0m 
>>> (in 0:01:51) [0m
>>>  [31mError while running thread
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
>>> _ret_via_queue
>>> queue.put({'return': func()})
>>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
>>> _deploy_host
>>> host.name (),
>>> RuntimeError: 
>>> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>>>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>>>  [36m  # Deploy environment:  [31mERROR [0m (in 0:01:51) [0m
>>>  [36m@ Deploy oVirt environment:  [31mERROR [0m (in 0:01:51) [0m
>>>  [31mError occured, aborting
>>> Traceback (most recent call last):
>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in 
>>> do_run
>>> self.cli_plugins[args.ovirtverb].do_run(args)
>>>   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in 
>>> do_run
>>> self._do_run(**vars(args))
>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in 
>>> wrapper
>>> return func(*args, **kwargs)
>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in 
>>> wrapper
>>> return func(*args, prefix=prefix, **kwargs)
>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in 
>>> do_deploy
>>> prefix.deploy()
>>>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
>>> wrapper
>>> return func(*args, **kwargs)
>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 67, 
>>> in wrapper
>>> return func(*args, **kwargs)
>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, 
>>> in deploy
>>> return super(OvirtPrefix, self).deploy()
>>>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
>>> wrapper
>>> return func(*args, **kwargs)
>>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in 
>>> deploy
>>> self._deploy_host, self.virt_env.get_vms().values()
>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in 
>>> invoke_in_parallel
>>> vt.join_all()
>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
>>> _ret_via_queue
>>> queue.put({'return': func()})
>>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
>>> _deploy_host
>>> host.name (),
>>> RuntimeError: 
>>> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>>>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>>> *
>>>
>>> [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>>> 4.0/3416/console
>>>
>>> Could you please check this issue?
>>>
>>>
>>> Thanks in advanced,
>>>
>>> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
>>> RHCSA | RHCE
>>> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>>>
>>> OPEN SOURCE - 1 4 011 && 011 4 1
>>>
>>> ___
>>> Devel mailing list
>>> 

Re: [ovirt-devel] test-repo_ovirt_experimental_4.0

[Eyal Edri]

I triggered the sdk job [1] so it should publish the rpms to 4.0 repo soon,
Lets give it one hour and see if it fixed the issue.

Regadless, please open a ticket in JIRA to investigate why the rpm was
deleted.

[1]
http://jenkins.ovirt.org/job/ovirt-engine-sdk_4.0_build-artifacts-el7-x86_64/73/console

On Wed, Dec 7, 2016 at 12:46 PM, Eyal Edri  wrote:

> try triggering the ovirt sdk job build artifacts for 4.0.
> maybe for some reason it was removed from experimental,  we will need to
> check what caused it to be removed.
>
> On Dec 7, 2016 12:39, "Shlomo Ben David"  wrote:
>
>> Hi All,
>>
>> The [1] job is failing with the following error:
>>
>> *+ yum install --nogpgcheck -y --downloaddir=/dev/shm ovirt-engine 
>> ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*'
>> Error: Package: 
>> ovirt-engine-cli-3.6.9.3-0.1.20161202.git61c5da9.el7.centos.noarch 
>> (alocalsync)
>>Requires: ovirt-engine-sdk-python >= 3.6.0.0
>>  [0m
>>  [36m* [Thread-4] Deploy VM lago-basic-suite-4-0-engine:  [31mERROR [0m 
>> (in 0:01:51) [0m
>>  [31mError while running thread
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
>> _ret_via_queue
>> queue.put({'return': func()})
>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
>> _deploy_host
>> host.name (),
>> RuntimeError: 
>> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>>  [36m  # Deploy environment:  [31mERROR [0m (in 0:01:51) [0m
>>  [36m@ Deploy oVirt environment:  [31mERROR [0m (in 0:01:51) [0m
>>  [31mError occured, aborting
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 264, in 
>> do_run
>> self.cli_plugins[args.ovirtverb].do_run(args)
>>   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in 
>> do_run
>> self._do_run(**vars(args))
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper
>> return func(*args, prefix=prefix, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in 
>> do_deploy
>> prefix.deploy()
>>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
>> wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 67, 
>> in wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, 
>> in deploy
>> return super(OvirtPrefix, self).deploy()
>>   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in 
>> wrapper
>> return func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in 
>> deploy
>> self._deploy_host, self.virt_env.get_vms().values()
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in 
>> invoke_in_parallel
>> vt.join_all()
>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in 
>> _ret_via_queue
>> queue.put({'return': func()})
>>   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in 
>> _deploy_host
>> host.name (),
>> RuntimeError: 
>> /home/jenkins/workspace/test-repo_ovirt_experimental_4.0/ovirt-system-tests/deployment-basic-suite-4.0/default/scripts/_home_jenkins_workspace_test-repo_ovirt_experimental_4.0_ovirt-system-tests_basic-suite-4.0_.._common_deploy-scripts_setup_engine.sh
>>  failed with status 1 on lago-basic-suite-4-0-engine [0m
>> *
>>
>> [1] - http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>> 4.0/3416/console
>>
>> Could you please check this issue?
>>
>>
>> Thanks in advanced,
>>
>> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
>> RHCSA | RHCE
>> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>>
>> OPEN SOURCE - 1 4 011 && 011 4 1
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>


-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Yaniv Kaul]

On Wed, Dec 7, 2016 at 10:57 AM, Roy Golan  wrote:

> Hi all,
>
> This is a discussion on the RFE[1] to provide a tool to perform full
> vacuum on our DBs.
>
> First if you are not familiar with vacuum please read this [2]
>
> # Backgroud
> ovirt 'engine' DB have several busy table with 2 differnt usage patten.
> One is audit_log and the others are the 'v*_statistics' tables and the
> difference between them is mostly inserts vs mostly hot updates.
> Tables with tons of updates creates garbage or 'dead' records that should
> be removed, and for this postgres have the aforementioned autovacuum
> cleaner. It will make the db reuse its already allocated space to perform
> future updates/inserts and so on.
> Autovacuum is essential for a db to function optimally and tweaking it is
> out of the scope of the feature.
>
> Full vacuum is designed to reclaim the disk space and reset the table
> statistics. It is a heavy maintenance task, it takes an exclusive lock on
> the table and may take seconds to minutes. In some situations it is
> effectively a downtime due to the long table lock and should not be running
> when the engine is running.
>
> # Critiria
> Provide a way to reclaim disk space claimed by the garbage created over
> time by the engine db and dwh.
>
> # Usage
> Either use it as part of the upgrade procedure (after all dbscipts
> execution)
> or just provide the tool and admin will run in on demand
> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
> - invocation:
>  ```
>  tool: [dbname(default engine)] [table: (default all)]
>  ```
> - if we invoke it on upgrade than an installation plugin should be added
> to invoke with default, no interaction
> - since VACUUM ANALYZE is consider a recommended maintenance task we can
> to it by default and ask the user for FULL.
>

Will the user know to answer intelligently if vacuum is needed or not?
Except for 'yes, you need it', we cannot even provide a time estimate (I
assume a disk space estimate is available!)
I would suggest to run ANALYZE for sure and provide an option at the end of
installation, to run the required command line - so make it as accessible
as possible, but not part of the flow.

I'm wondering if the community can run ANALYZE on their database, and we
can estimate how many are in dire need for full vacuum already.
Y.

- remote db is supported as well, doesn't have to be local
>
> # Questions
>  - Will remote dwh have the credentials under
> /etc/ovirt-engine/engine.conf.d?
>  - Should  AAA schema be taken into account as well?
>
> Please review, thanks
> Roy
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430
> [2] https://www.postgresql.org/docs/9.2/static/runtime-
> config-autovacuum.html
> [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html
>
> ___
> Users mailing list
> us...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Yedidyah Bar David]

On Wed, Dec 7, 2016 at 12:08 PM, Piotr Kliczewski
 wrote:
> On Wed, Dec 7, 2016 at 9:57 AM, Roy Golan  wrote:
>> Hi all,
>>
>> This is a discussion on the RFE[1] to provide a tool to perform full vacuum
>> on our DBs.
>>
>> First if you are not familiar with vacuum please read this [2]
>>
>> # Backgroud
>> ovirt 'engine' DB have several busy table with 2 differnt usage patten. One
>> is audit_log and the others are the 'v*_statistics' tables and the
>> difference between them is mostly inserts vs mostly hot updates.
>> Tables with tons of updates creates garbage or 'dead' records that should be
>> removed, and for this postgres have the aforementioned autovacuum cleaner.
>> It will make the db reuse its already allocated space to perform future
>> updates/inserts and so on.
>> Autovacuum is essential for a db to function optimally and tweaking it is
>> out of the scope of the feature.
>>
>> Full vacuum is designed to reclaim the disk space and reset the table
>> statistics. It is a heavy maintenance task, it takes an exclusive lock on
>> the table and may take seconds to minutes. In some situations it is
>> effectively a downtime due to the long table lock and should not be running
>> when the engine is running.
>>
>> # Critiria
>> Provide a way to reclaim disk space claimed by the garbage created over time
>> by the engine db and dwh.
>
> What about not storing this data in db? Do we need it all the time or
> just for some amount of time?
>
>>
>> # Usage
>> Either use it as part of the upgrade procedure (after all dbscipts
>> execution)
>> or just provide the tool and admin will run in on demand
>> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
>> - invocation:
>>  ```
>>  tool: [dbname(default engine)] [table: (default all)]
>>  ```
>> - if we invoke it on upgrade than an installation plugin should be added to
>> invoke with default, no interaction
>> - since VACUUM ANALYZE is consider a recommended maintenance task we can to
>> it by default and ask the user for FULL.
>> - remote db is supported as well, doesn't have to be local
>>
>> # Questions
>>  - Will remote dwh have the credentials under
>> /etc/ovirt-engine/engine.conf.d?

The engine needs access to the dwh db. The credentials the engine uses
are saved in above directory.

dwhd also needs access to its db. The credentials it uses are saved in:

/etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/

The keys (DWH_DB_HOST, DWH_DB_PORT etc.) are the same for both.

If dwh is a separate machine, you'll have the latter only on that
machine.

If the scope of current project does include dwh, it should consider
both same and different machines for engine+dwh. E.g., when you full-
vacuum the DBs (each of them separately), you effectively suspend both
engine and dwhd (if indeed also dwhd - did you check?), which can be
on two different machines. So:

1. If you run this tool standalone, on the engine machine, for both DBs,
you have to check also how it affects dwhd and handle.

2. If you run this during engine-setup on the engine machine, for both
DBs, it already asks dwhd to stop, even if on a remote machine. There is
a cronjob/systemd conf that should restart it once an hour.

3. If you want to provide a separate tool for dwh db, and allow running
it on the dwh machine, you have to handle the engine.

>>  - Should  AAA schema be taken into account as well?
>>
>> Please review, thanks
>> Roy
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430
>> [2]
>> https://www.postgresql.org/docs/9.2/static/runtime-config-autovacuum.html
>> [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html
>>
>> ___
>> Users mailing list
>> us...@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel



-- 
Didi
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [Feture discussion] Full vacuum tool

[Piotr Kliczewski]

On Wed, Dec 7, 2016 at 9:57 AM, Roy Golan  wrote:
> Hi all,
>
> This is a discussion on the RFE[1] to provide a tool to perform full vacuum
> on our DBs.
>
> First if you are not familiar with vacuum please read this [2]
>
> # Backgroud
> ovirt 'engine' DB have several busy table with 2 differnt usage patten. One
> is audit_log and the others are the 'v*_statistics' tables and the
> difference between them is mostly inserts vs mostly hot updates.
> Tables with tons of updates creates garbage or 'dead' records that should be
> removed, and for this postgres have the aforementioned autovacuum cleaner.
> It will make the db reuse its already allocated space to perform future
> updates/inserts and so on.
> Autovacuum is essential for a db to function optimally and tweaking it is
> out of the scope of the feature.
>
> Full vacuum is designed to reclaim the disk space and reset the table
> statistics. It is a heavy maintenance task, it takes an exclusive lock on
> the table and may take seconds to minutes. In some situations it is
> effectively a downtime due to the long table lock and should not be running
> when the engine is running.
>
> # Critiria
> Provide a way to reclaim disk space claimed by the garbage created over time
> by the engine db and dwh.

What about not storing this data in db? Do we need it all the time or
just for some amount of time?

>
> # Usage
> Either use it as part of the upgrade procedure (after all dbscipts
> execution)
> or just provide the tool and admin will run in on demand
> - engine db credentials read from /etc/ovirt-engine/engine.conf.d/
> - invocation:
>  ```
>  tool: [dbname(default engine)] [table: (default all)]
>  ```
> - if we invoke it on upgrade than an installation plugin should be added to
> invoke with default, no interaction
> - since VACUUM ANALYZE is consider a recommended maintenance task we can to
> it by default and ask the user for FULL.
> - remote db is supported as well, doesn't have to be local
>
> # Questions
>  - Will remote dwh have the credentials under
> /etc/ovirt-engine/engine.conf.d?
>  - Should  AAA schema be taken into account as well?
>
> Please review, thanks
> Roy
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430
> [2]
> https://www.postgresql.org/docs/9.2/static/runtime-config-autovacuum.html
> [3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html
>
> ___
> Users mailing list
> us...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] [Feture discussion] Full vacuum tool

[Roy Golan]

Hi all,

This is a discussion on the RFE[1] to provide a tool to perform full vacuum
on our DBs.

First if you are not familiar with vacuum please read this [2]

# Backgroud
ovirt 'engine' DB have several busy table with 2 differnt usage patten. One
is audit_log and the others are the 'v*_statistics' tables and the
difference between them is mostly inserts vs mostly hot updates.
Tables with tons of updates creates garbage or 'dead' records that should
be removed, and for this postgres have the aforementioned autovacuum
cleaner. It will make the db reuse its already allocated space to perform
future updates/inserts and so on.
Autovacuum is essential for a db to function optimally and tweaking it is
out of the scope of the feature.

Full vacuum is designed to reclaim the disk space and reset the table
statistics. It is a heavy maintenance task, it takes an exclusive lock on
the table and may take seconds to minutes. In some situations it is
effectively a downtime due to the long table lock and should not be running
when the engine is running.

# Critiria
Provide a way to reclaim disk space claimed by the garbage created over
time by the engine db and dwh.

# Usage
Either use it as part of the upgrade procedure (after all dbscipts
execution)
or just provide the tool and admin will run in on demand
- engine db credentials read from /etc/ovirt-engine/engine.conf.d/
- invocation:
 ```
 tool: [dbname(default engine)] [table: (default all)]
 ```
- if we invoke it on upgrade than an installation plugin should be added to
invoke with default, no interaction
- since VACUUM ANALYZE is consider a recommended maintenance task we can to
it by default and ask the user for FULL.
- remote db is supported as well, doesn't have to be local

# Questions
 - Will remote dwh have the credentials under
/etc/ovirt-engine/engine.conf.d?
 - Should  AAA schema be taken into account as well?

Please review, thanks
Roy

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1388430
[2]
https://www.postgresql.org/docs/9.2/static/runtime-config-autovacuum.html
[3] https://www.postgresql.org/docs/devel/static/sql-vacuum.html
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Correct implementation of virt-sysprep job

[Oved Ourfali]

On Tue, Dec 6, 2016 at 11:12 PM, Adam Litke  wrote:

> On 06/12/16 22:06 +0200, Arik Hadas wrote:
>
>> Adam,
>>
>
> :)  You seem upset.  Sorry if I touched on a nerve...
>
> Just out of curiosity: when you write "v2v has promised" - what exactly do
>> you
>> mean? the tool? Richard Jones (the maintainer of virt-v2v)? Shahar and I
>> that
>> implemented the integration with virt-v2v? I'm not aware of such a
>> promise by
>> any of these options :)
>>
>
> Some history...
>
> Earlier this year Nir, Francesco (added), Shahar, and I began
> discussing the similarities between what storage needed to do with
> external commands and what was designed specifically for v2v.  I am
> not sure if you were involved in the project at that time.  The plan
> was to create common infrastructure that could be extended to fit the
> unique needs of the verticals.  The v2v code was going to be moved
> over to the new infrastructure (see [1]) and the only thing that
> stopped the initial patch was lack of a VMWare testing environment for
> verification.
>
> At that time storage refocused on developing verbs that used the new
> infrastructure and have been maintaining its suitability for general
> use.  Conversion of v2v -> Host Jobs is obviously a lower priority
> item and much more difficult now due to the early missed opportunity.
>
> Anyway, let's say that you were given such a promise by someone and thus
>> consider that mechanism to be deprecated - it doesn't really matter.
>>
>
> I may be biased but I think my opinion does matter.
>
> The current implementation doesn't well fit to this flow (it requires
>> per-volume job, it creates leases that are not needed for template's
>> disks,
>> ...) and with the "next-gen API" with proper support for virt flows not
>> even
>> being discussed with us (and iiuc also not with the infra team) yet, I
>> don't
>> understand what do you suggest except for some strong, though irrelevant,
>> statements.
>>
>
> If you are willing to engage in a good-faith technical discussion I am
> sure I can help you to understand.  These operations to storage demand
> some form of locking protection.  If volume leases aren't appropriate then
> perhaps we should use the VM Leases / xleases that Nir is finishing
> off for 4.1 now.
>
> I suggest loud and clear to reuse (not to add dependencies, not to
>> enhance, ..)
>> an existing mechanism for a very similar flow of virt-v2v that works well
>> and
>> simple.
>>
>
> I clearly remember discussions involving infra (hello Oved), virt
> (hola Michal), and storage where we decided that new APIs performing
> async operations involving external commands should use the HostJobs
> infrastructure instead of adding more information to Host Stats.
> These were the "famous" entity polling meetings.
>
> Of course plans can change but I have never been looped into any such
> discussions.
>
>
Well, I think that when someone builds a good infrastructure he first needs
to talk to all consumers and make sure it fits.
In this case it seems like most work was done to fit the storage use-case,
and now you check whether it can fit others as well

IMO it makes much more sense to use events where possible (and you've
promised to use those as well, but I don't see you doing that...). v2v
should use events for sure, and they have promised to do that in the past,
instead of using the v2v jobs. The reason events weren't used originally
with the v2v feature, was that it was too risky and the events
infrastructure was added too late in the game.



> Do you "promise" to implement your "next gen API" for 4.1 as an
>> alternative?
>>
>
> I guess we need the design first.
>
>
> On Tue, Dec 6, 2016 at 5:04 PM, Adam Litke  wrote:
>>
>>On 05/12/16 11:17 +0200, Arik Hadas wrote:
>>
>>
>>
>>On Mon, Dec 5, 2016 at 10:05 AM, Nir Soffer 
>> wrote:
>>
>>   On Sun, Dec 4, 2016 at 8:50 PM, Shmuel Melamud <
>> smela...@redhat.com>
>>wrote:
>>   >
>>   > Hi!
>>   >
>>   > I'm currently working on integration of virt-sysprep into
>> oVirt.
>>   >
>>   > Usually, if user creates a template from a regular VM, and
>> then
>>creates
>>   new VMs from this template, these new VMs inherit all
>> configuration
>>of the
>>   original VM, including SSH keys, UDEV rules, MAC addresses,
>> system
>>ID,
>>   hostname etc. It is unfortunate, because you cannot have two
>> network
>>   devices with the same MAC address in the same network, for
>> example.
>>   >
>>   > To avoid this, user must clean all machine-specific
>> configuration
>>from
>>   the original VM before creating a template from it. You can do
>> this
>>   manually, but there is virt-sysprep utility that does this
>>automatically.
>>   >
>>   > Ideally, virt-sysprep should be seamlessly