[ovirt-devel] Re: Running Vdsm on Python 3

[Marcin Sobczyk]

Hi,

On 2/10/19 10:41 AM, Gal Zaidman wrote:



On Fri, Feb 8, 2019 at 10:42 PM Milan Zamazal > wrote:


Dan Kenigsberg mailto:dan...@redhat.com>> writes:

> On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal
mailto:mzama...@redhat.com>> wrote:
>>
>> Hi, there have been efforts by different Vdsm developers to
port Vdsm to
>
>> Python 3.  I've joined the efforts recently and I focused on
the aim to
>> make Vdsm running (not necessarily working correctly initially) on
>> Python 3, in order to better understand all the related
problems and to
>> test Python 3 compatibility (among other) on a running Vdsm
instance.
>>
>> I can currently start Vdsm on Python 3, although it can't
communicate
>> with clients yet (such as Engine), see below.  FWIW, here is
what I had
>> to do to make Vdsm at least installable and startable with
Python 3.
>>
>> I use Fedora 28 as the platform.  The primary reason is that
ovirt.org 
>> provides for Fedora 28 all the non-Vdsm oVirt packages needed
to run
>> Vdsm on the host.
>>
>> Direct host installation doesn't work for me yet with Python 3.  It
>> might be a nice next goal but there is an added complexity in
requiring
>> proper support in all the related oVirt packages, not just
Vdsm.  It's
>> better to have a basically working Vdsm first.  So I had
installed the
>> host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.


I remember that I once tried installing the engine with python3, and 
adding an fc28 host worked fine, what failed when you tried it ?


>> One non-Vdsm Python package required by Vdsm and not yet ported to
>> Python 3 is sanlock-python.  There is an ongoing work on Python
3 port
>> at
https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3.
>> I didn't know about it so I used my own hack that made sanlock
Python
>> module at least importable on Python 3. Let's discuss it on
>> sanlock-devel.
>>
>> Regarding Vdsm itself, there are several areas to fix.  First,
there are
>> compatibility problems in Python code.  I started making some
fixes in a
>> patch series starting with https://gerrit.ovirt.org/97019.
Please note
>> there are other Python 3 patches by other Vdsm developers.  I
hope that
>> once we branch out 4.3, we can start merging the patches to master
>> quickly and better coordinate our efforts.
>>
>> Then there are other changes to Vdsm needed, which I don't have
patches
>> for (just my ugly private hacks):
>
> I believe that +Gal Zaidman has few patches for these in his drawer.
> He has postponed his work due to 4.3 stabilitzation.

Yes.  Gal, perhaps you can work on making the patches ready in the
meantime, so that they can be merged quickly once we make 4.3 branch?


I currently understand that in terms of packaging we have to do some 
stuff before we can add the python2/3 packages in VDSM:
1. Clean the spec %files section: I send a patch: 
https://gerrit.ovirt.org/#/c/97266/ which starts the cleaning process, 
I had some questions there to see if we like the direction of this 
patch, would be happy to get your opinion on it :)
2. Agree on a way to package without duplicating code, I had this 
patch https://gerrit.ovirt.org/#/c/96996/ which started the 
discussion, I think that a good approach is to use Jinja templeting as 
a layer on top of the spec.in  instead of the script 
that I added,  maybe even have a python-vdsm-packages.in 
 file which we can use to generate the 
%packages, I will work on a patch for that soon.


>> - All the /usr/bin/python2 shebangs must be replaced with
>>   /usr/bin/python3.  We should find a good way to solve and
automated
>>   that, perhaps as part of the installation process depending
on Python
>>   version selected.

Regarding python2/python3 shebangs - unfortunately the most sensible 
solution that comes to my mind is:


- move as much logic as possible away from shebanged scripts to libraries
- rename the shebanged scripts to "script.py.in"
- replace the python version in the shebang with m4 when building

Changing python version as part of installation process is not a good 
idea IMO.



>>
>> - configure.ac  must be adjusted:
>>
>>   + AM_PATH_PYTHON([3.6])
>>   + AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>>   + AX_PYTHON_MODULE([six], [fatal], python3)
>>
>>   It should be easy to make a patch for that.


I think that we should remove most of the python2/3 stuff from the 
configure.ac  such as PYTHON3_SUPPORT, fedora_run 
and all those stuff and decide which python version we are using in 
the spec file based on the OS and pass it as a para

[ovirt-devel] Re: Running Vdsm on Python 3

[Gal Zaidman]

On Fri, Feb 8, 2019 at 10:42 PM Milan Zamazal  wrote:

> Dan Kenigsberg  writes:
>
> > On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal 
> wrote:
> >>
> >> Hi, there have been efforts by different Vdsm developers to port Vdsm to
> >
> >> Python 3.  I've joined the efforts recently and I focused on the aim to
> >> make Vdsm running (not necessarily working correctly initially) on
> >> Python 3, in order to better understand all the related problems and to
> >> test Python 3 compatibility (among other) on a running Vdsm instance.
> >>
> >> I can currently start Vdsm on Python 3, although it can't communicate
> >> with clients yet (such as Engine), see below.  FWIW, here is what I had
> >> to do to make Vdsm at least installable and startable with Python 3.
> >>
> >> I use Fedora 28 as the platform.  The primary reason is that ovirt.org
> >> provides for Fedora 28 all the non-Vdsm oVirt packages needed to run
> >> Vdsm on the host.
> >>
> >> Direct host installation doesn't work for me yet with Python 3.  It
> >> might be a nice next goal but there is an added complexity in requiring
> >> proper support in all the related oVirt packages, not just Vdsm.  It's
> >> better to have a basically working Vdsm first.  So I had installed the
> >> host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
>
>
I remember that I once tried installing the engine with python3, and adding
an fc28 host worked fine, what failed when you tried it ?


> >> One non-Vdsm Python package required by Vdsm and not yet ported to
> >> Python 3 is sanlock-python.  There is an ongoing work on Python 3 port
> >> at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3
> .
> >> I didn't know about it so I used my own hack that made sanlock Python
> >> module at least importable on Python 3.  Let's discuss it on
> >> sanlock-devel.
> >>
> >> Regarding Vdsm itself, there are several areas to fix.  First, there are
> >> compatibility problems in Python code.  I started making some fixes in a
> >> patch series starting with https://gerrit.ovirt.org/97019.  Please note
> >> there are other Python 3 patches by other Vdsm developers.  I hope that
> >> once we branch out 4.3, we can start merging the patches to master
> >> quickly and better coordinate our efforts.
> >>
> >> Then there are other changes to Vdsm needed, which I don't have patches
> >> for (just my ugly private hacks):
> >
> > I believe that +Gal Zaidman has few patches for these in his drawer.
> > He has postponed his work due to 4.3 stabilitzation.
>
> Yes.  Gal, perhaps you can work on making the patches ready in the
> meantime, so that they can be merged quickly once we make 4.3 branch?
>

I currently understand that in terms of packaging we have to do some stuff
before we can add the python2/3 packages in VDSM:
1. Clean the spec %files section: I send a patch:
https://gerrit.ovirt.org/#/c/97266/ which starts the cleaning process, I
had some questions there to see if we like the direction of this patch,
would be happy to get your opinion on it :)
2. Agree on a way to package without duplicating code, I had this patch
https://gerrit.ovirt.org/#/c/96996/ which started the discussion, I think
that a good approach is to use Jinja templeting as a layer on top of the
spec.in instead of the script that I added,  maybe even have a
python-vdsm-packages.in file which we can use to generate the %packages, I
will work on a patch for that soon.


> >> - All the /usr/bin/python2 shebangs must be replaced with
> >>   /usr/bin/python3.  We should find a good way to solve and automated
> >>   that, perhaps as part of the installation process depending on Python
> >>   version selected.
> >>
> >> - configure.ac must be adjusted:
> >>
> >>   + AM_PATH_PYTHON([3.6])
> >>   + AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
> >>   + AX_PYTHON_MODULE([six], [fatal], python3)
> >>
> >>   It should be easy to make a patch for that.
>

I think that we should remove most of the python2/3 stuff from the
configure.ac such as PYTHON3_SUPPORT, fedora_run and all those stuff and
decide which python version we are using in the spec file based on the OS
and pass it as a parameter to the configure.


> >> - The following changes in vdsm.spec.in are needed:
> >>
> >>   + Dependencies on Python packages must be adjusted.
> >>   + python3_sitelib must be used instead of python_sitelib for Python 3.
> >>   + Byte compiled files are placed into __pycache__ subdirectories by
> >> Python 3 and they must be installed.
> >>
>

we will do all that per package once we will start packaging


> >>   It would be nice to have patches for that so that we can build
> >>   Python 3 rpm's easily.
> >>
> >> That's it at the moment.  Most compatibility problems in Python code are
> >> easy to fix once they are identified.  One important exception is bytes
> >> versus strings -- that's going to be delicate.
> >>
> >> The bytes versus strings problem is very present in communication
> >> code -- such as vdsm/rp

[ovirt-devel] Re: Running Vdsm on Python 3

[Milan Zamazal]

Dan Kenigsberg  writes:

> On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal  wrote:
>>
>> Hi, there have been efforts by different Vdsm developers to port Vdsm to
>
>> Python 3.  I've joined the efforts recently and I focused on the aim to
>> make Vdsm running (not necessarily working correctly initially) on
>> Python 3, in order to better understand all the related problems and to
>> test Python 3 compatibility (among other) on a running Vdsm instance.
>>
>> I can currently start Vdsm on Python 3, although it can't communicate
>> with clients yet (such as Engine), see below.  FWIW, here is what I had
>> to do to make Vdsm at least installable and startable with Python 3.
>>
>> I use Fedora 28 as the platform.  The primary reason is that ovirt.org
>> provides for Fedora 28 all the non-Vdsm oVirt packages needed to run
>> Vdsm on the host.
>>
>> Direct host installation doesn't work for me yet with Python 3.  It
>> might be a nice next goal but there is an added complexity in requiring
>> proper support in all the related oVirt packages, not just Vdsm.  It's
>> better to have a basically working Vdsm first.  So I had installed the
>> host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
>>
>> One non-Vdsm Python package required by Vdsm and not yet ported to
>> Python 3 is sanlock-python.  There is an ongoing work on Python 3 port
>> at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3.
>> I didn't know about it so I used my own hack that made sanlock Python
>> module at least importable on Python 3.  Let's discuss it on
>> sanlock-devel.
>>
>> Regarding Vdsm itself, there are several areas to fix.  First, there are
>> compatibility problems in Python code.  I started making some fixes in a
>> patch series starting with https://gerrit.ovirt.org/97019.  Please note
>> there are other Python 3 patches by other Vdsm developers.  I hope that
>> once we branch out 4.3, we can start merging the patches to master
>> quickly and better coordinate our efforts.
>>
>> Then there are other changes to Vdsm needed, which I don't have patches
>> for (just my ugly private hacks):
>
> I believe that +Gal Zaidman has few patches for these in his drawer.
> He has postponed his work due to 4.3 stabilitzation.

Yes.  Gal, perhaps you can work on making the patches ready in the
meantime, so that they can be merged quickly once we make 4.3 branch?

>> - All the /usr/bin/python2 shebangs must be replaced with
>>   /usr/bin/python3.  We should find a good way to solve and automated
>>   that, perhaps as part of the installation process depending on Python
>>   version selected.
>>
>> - configure.ac must be adjusted:
>>
>>   + AM_PATH_PYTHON([3.6])
>>   + AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>>   + AX_PYTHON_MODULE([six], [fatal], python3)
>>
>>   It should be easy to make a patch for that.
>>
>> - The following changes in vdsm.spec.in are needed:
>>
>>   + Dependencies on Python packages must be adjusted.
>>   + python3_sitelib must be used instead of python_sitelib for Python 3.
>>   + Byte compiled files are placed into __pycache__ subdirectories by
>> Python 3 and they must be installed.
>>
>>   It would be nice to have patches for that so that we can build
>>   Python 3 rpm's easily.
>>
>> That's it at the moment.  Most compatibility problems in Python code are
>> easy to fix once they are identified.  One important exception is bytes
>> versus strings -- that's going to be delicate.
>>
>> The bytes versus strings problem is very present in communication
>> code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc.  While strings
>> constants are typically used in the code, the communication means work
>> with bytes.  This mismatch prevents Vdsm from communicating with the
>> clients, so it would be very useful if someone could look at it and fix
>> the code (one of the harder parts would probably be to decide where to
>> keep bytes and where to already convert them to strings).
>>
>> HTH.  As more people are involved in porting Vdsm to Python 3, it might
>> be useful to coordinate all the efforts here, to avoid duplicate work,
>> to share findings and to speed up the process.
>
> Indeed. We may also try to revive Trello usage.
> Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?

Sure, a couple of tasks added.
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OGPVUPLQ2P5XQWSOY345P5WG7MKIIV4O/

[ovirt-devel] Re: Running Vdsm on Python 3

[Dan Kenigsberg]

On Thu, Feb 7, 2019 at 11:04 PM Milan Zamazal  wrote:
>
> Hi, there have been efforts by different Vdsm developers to port Vdsm to
> Python 3.  I've joined the efforts recently and I focused on the aim to
> make Vdsm running (not necessarily working correctly initially) on
> Python 3, in order to better understand all the related problems and to
> test Python 3 compatibility (among other) on a running Vdsm instance.
>
> I can currently start Vdsm on Python 3, although it can't communicate
> with clients yet (such as Engine), see below.  FWIW, here is what I had
> to do to make Vdsm at least installable and startable with Python 3.
>
> I use Fedora 28 as the platform.  The primary reason is that ovirt.org
> provides for Fedora 28 all the non-Vdsm oVirt packages needed to run
> Vdsm on the host.
>
> Direct host installation doesn't work for me yet with Python 3.  It
> might be a nice next goal but there is an added complexity in requiring
> proper support in all the related oVirt packages, not just Vdsm.  It's
> better to have a basically working Vdsm first.  So I had installed the
> host with Python 2 Vdsm and then upgraded Vdsm to Python 3 version.
>
> One non-Vdsm Python package required by Vdsm and not yet ported to
> Python 3 is sanlock-python.  There is an ongoing work on Python 3 port
> at https://pagure.io/fork/mcsontos/sanlock/commits/dev-mcsontos-python3.
> I didn't know about it so I used my own hack that made sanlock Python
> module at least importable on Python 3.  Let's discuss it on
> sanlock-devel.
>
> Regarding Vdsm itself, there are several areas to fix.  First, there are
> compatibility problems in Python code.  I started making some fixes in a
> patch series starting with https://gerrit.ovirt.org/97019.  Please note
> there are other Python 3 patches by other Vdsm developers.  I hope that
> once we branch out 4.3, we can start merging the patches to master
> quickly and better coordinate our efforts.
>
> Then there are other changes to Vdsm needed, which I don't have patches
> for (just my ugly private hacks):

I believe that +Gal Zaidman has few patches for these in his drawer.
He has postponed his work due to 4.3 stabilitzation.


>
> - All the /usr/bin/python2 shebangs must be replaced with
>   /usr/bin/python3.  We should find a good way to solve and automated
>   that, perhaps as part of the installation process depending on Python
>   version selected.
>
> - configure.ac must be adjusted:
>
>   + AM_PATH_PYTHON([3.6])
>   + AC_PATH_PROG([PYTHON_CONFIG], [python3-config])
>   + AX_PYTHON_MODULE([six], [fatal], python3)
>
>   It should be easy to make a patch for that.
>
> - The following changes in vdsm.spec.in are needed:
>
>   + Dependencies on Python packages must be adjusted.
>   + python3_sitelib must be used instead of python_sitelib for Python 3.
>   + Byte compiled files are placed into __pycache__ subdirectories by
> Python 3 and they must be installed.
>
>   It would be nice to have patches for that so that we can build
>   Python 3 rpm's easily.
>
> That's it at the moment.  Most compatibility problems in Python code are
> easy to fix once they are identified.  One important exception is bytes
> versus strings -- that's going to be delicate.
>
> The bytes versus strings problem is very present in communication
> code -- such as vdsm/rpc/http.py, vdsm/sslutils.py, etc.  While strings
> constants are typically used in the code, the communication means work
> with bytes.  This mismatch prevents Vdsm from communicating with the
> clients, so it would be very useful if someone could look at it and fix
> the code (one of the harder parts would probably be to decide where to
> keep bytes and where to already convert them to strings).
>
> HTH.  As more people are involved in porting Vdsm to Python 3, it might
> be useful to coordinate all the efforts here, to avoid duplicate work,
> to share findings and to speed up the process.

Indeed. We may also try to revive Trello usage.
Would you add relevant tasks to https://trello.com/b/tHeO1BFW/py3-vdsm ?


>
> Thanks,
> Milan
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MRGBB2VJJX6DTUJ65UYEGAWFFGDW55VQ/
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UXN6VLKOOUUY7NZM5PKZHXMTOVQ5DOZB/