Re: [PATCH v2 03/15] python: add VERSION file
On Tue, 2020-10-20 at 10:06 +0100, Daniel P. Berrangé wrote: > The QEMU python modules are not like other python modules though, > precisely because they are talking to QEMU. If we are shipping > QEMU python releases on the same schedule as QEMU, then we can > expect the normal ase to be updating both QEMU & Python together. Once you start uploading the Python packages to PyPi, you really have no way to ensure this will be the case. -- Andrea Bolognani / Red Hat / Virtualization
Re: [PATCH v2 03/15] python: add VERSION file
On Tue, Oct 20, 2020 at 10:52:14AM +0200, Andrea Bolognani wrote: > On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote: > > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > > > With that in mind, I think it would be unwise for qemu.* not to do > > > the same; in particular, using a version number that's not <1.0.0 for > > > a package that is very much in flux will almost certainly break > > > people's expectations, and is also not something that you can easily > > > take back at a later time. > > > > I don't think it is that big a deal, and there is clear benefit to > > having the python code version match the QEMU version that it is > > the companioon to. > > > > Ultimately the versioning scheme just impacts on the version string > > conditionals people list for their dependancies. Apps consuming QEMU > > can handle any of the version schemes without much difference. > > The problem comes from the expectations: a Python programmer, who is > used to semver due to its prominence on PyPi, when deciding whether > to move from qemu.core 4.2.0 to 5.0.0 might expect to need code > changes to cope with API-breaking changes - where in fact there are > none, and at the same time might expect upgrading to 5.2.0 from 5.0.0 > to be completely straightforward when in reality a feature their > application depends on might have been removed after the usual > deprecation period. The QEMU python modules are not like other python modules though, precisely because they are talking to QEMU. If we are shipping QEMU python releases on the same schedule as QEMU, then we can expect the normal ase to be updating both QEMU & Python together. So regardless of versioning in the python code, the QMP code they are talking to is liable to have removed deprecated features they are using. IMHO the upgrade issue is largely a problem of docs and testing, semver is no magic bullet. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH v2 03/15] python: add VERSION file
On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote: > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > > I think this need to be considered very carefully. > > > > I'm not overly familiar with the Python ecosystem but it would appear > > that, despite PEP 440 not mandating this, many (most?) of the > > packages uploaded to PyPi are using semantic versioning. > > > https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme > > Semver is the recommended approach, but they explicitly list date > based versioning as a valid alternative > > "Semantic versioning is not a suitable choice for all projects, >such as those with a regular time based release cadence and a >deprecation process that provides warnings for a number of >releases prior to removal of a feature." > > That paragraph describes QEMU's scenario. The section on date based versioning continues with A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular release is given just the version number. Version numbers for date based projects typically take the form of YEAR.MONTH (for example, 12.04, 15.10). The problem with QEMU's version numbers is that, while they are date based, they still *look* like semver, so it wouldn't be at all unreasonable for the user to expect that they also *behave* like semver. This is not much of a problem when it comes to the main binary, but it is certainly much more confusing when you start using the same version number for a Python library. > > With that in mind, I think it would be unwise for qemu.* not to do > > the same; in particular, using a version number that's not <1.0.0 for > > a package that is very much in flux will almost certainly break > > people's expectations, and is also not something that you can easily > > take back at a later time. > > I don't think it is that big a deal, and there is clear benefit to > having the python code version match the QEMU version that it is > the companioon to. > > Ultimately the versioning scheme just impacts on the version string > conditionals people list for their dependancies. Apps consuming QEMU > can handle any of the version schemes without much difference. The problem comes from the expectations: a Python programmer, who is used to semver due to its prominence on PyPi, when deciding whether to move from qemu.core 4.2.0 to 5.0.0 might expect to need code changes to cope with API-breaking changes - where in fact there are none, and at the same time might expect upgrading to 5.2.0 from 5.0.0 to be completely straightforward when in reality a feature their application depends on might have been removed after the usual deprecation period. -- Andrea Bolognani / Red Hat / Virtualization
Re: [PATCH v2 03/15] python: add VERSION file
On 10/19/20 6:02 AM, Daniel P. Berrangé wrote: On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible to my knowledge. The problem is that when using pip to build and install a python package, it copies files over to a temporary directory and performs its build there. This loses access to any information in the parent directory, including git itself. Further, Python versions have a standard (PEP 440) that may or may not follow QEMU's versioning. In general, it does; but naturally QEMU does not follow PEP 440. To avoid any automatically-generated conflict, a manual version file is preferred. I am proposing: - Python core tooling synchronizes with the QEMU version directly (5.2.0, 5.1.1, 5.3.0, etc.) - In the event that a Python package needs to be updated independently of the QEMU version, a pre-release alpha version should be preferred, but *only* after inclusion to the qemu development or stable branches. e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's release. - The Python core tooling makes absolutely no version compatibility checks or constraints. It *may* work with releases of QEMU from the past or future, but it is not required to. i.e., "qemu.core" will always remain in lock-step with QEMU. - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp and begin indepedently versioning such a package separately from the QEMU version it accompanies. I think this need to be considered very carefully. I'm not overly familiar with the Python ecosystem but it would appear that, despite PEP 440 not mandating this, many (most?) of the packages uploaded to PyPi are using semantic versioning. https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme Semver is the recommended approach, but they explicitly list date based versioning as a valid alternative "Semantic versioning is not a suitable choice for all projects, such as those with a regular time based release cadence and a deprecation process that provides warnings for a number of releases prior to removal of a feature." That paragraph describes QEMU's scenario. NB, historically we've made arbitrary changes to the python code since it was not considered public API. If we make it official public API, then we would actually need to start following our deprecation process for the python code too. I think our deprecation process is not tightly compatible with how Python programmers at-large expect packages to work. Semver is more or less the norm, despite the fact that it isn't explicitly required. setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc often hinge on a major version, e.g. qemu >= 5.0 qemu >= 3.0, < 6.0 would both be common forms of describing a requirement. Pinning specific versions is considered bad form, but in the context of releasing a package, I often see maintainers hedging their bets and preventing upgrades across a major version line. For that reason I am a little weary of adopting the deprecation policy as it exists in QEMU directly, and would propose a modification for my purposes here: - Features must be marked as deprecated - They must remain in a deprecated state for [at least] 2 releases - Deprecated features may not be removed until a major version change. In practice, this modification is a change from "2 releases" to "at least 2". However, I didn't intend to pay any mind to the deprecation policy "yet", as I have the package metadata listing the package status as "Alpha", see below: With that in mind, I think it would be unwise for qemu.* not to do the same; in particular, using a version number that's not <1.0.0 for a package that is very much in flux will almost certainly break people's expectations, and is also not something that you can easily take back at a later time. I don't think it is that big a deal, and there is clear benefit to having the python code version match the QEMU version that it is the companioon to. Do you think it's fine if I start versioning at, say, "0.5.2", I could ignore a deprecation policy for now? I have a lot of changes I want to make and expect a lot of breaking changes very quickly. I just wanted to try -- somehow -- to conjure up a relationship to the QEMU package it's designed to work with. Ultimately the versioning scheme just impacts on the version string conditionals people list for their dependancies. Apps consuming QEMU can handle any of the version schemes without much difference. Regards, Daniel Thanks for your input. This is the trickiest part of the process for me. I believe there is value in distributing
Re: [PATCH v2 03/15] python: add VERSION file
On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: > > Python infrastructure as it exists today is not capable reliably of > > single-sourcing a package version from a parent directory. The authors > > of pip are working to correct this, but as of today this is not possible > > to my knowledge. > > > > The problem is that when using pip to build and install a python > > package, it copies files over to a temporary directory and performs its > > build there. This loses access to any information in the parent > > directory, including git itself. > > > > Further, Python versions have a standard (PEP 440) that may or may not > > follow QEMU's versioning. In general, it does; but naturally QEMU does > > not follow PEP 440. To avoid any automatically-generated conflict, a > > manual version file is preferred. > > > > > > I am proposing: > > > > - Python core tooling synchronizes with the QEMU version directly > > (5.2.0, 5.1.1, 5.3.0, etc.) > > > > - In the event that a Python package needs to be updated independently > > of the QEMU version, a pre-release alpha version should be preferred, > > but *only* after inclusion to the qemu development or stable branches. > > > > e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's > > release. > > > > - The Python core tooling makes absolutely no version compatibility > > checks or constraints. It *may* work with releases of QEMU from the > > past or future, but it is not required to. > > > > i.e., "qemu.core" will always remain in lock-step with QEMU. > > > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp > > and begin indepedently versioning such a package separately from the > > QEMU version it accompanies. > > I think this need to be considered very carefully. > > I'm not overly familiar with the Python ecosystem but it would appear > that, despite PEP 440 not mandating this, many (most?) of the > packages uploaded to PyPi are using semantic versioning. https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme Semver is the recommended approach, but they explicitly list date based versioning as a valid alternative "Semantic versioning is not a suitable choice for all projects, such as those with a regular time based release cadence and a deprecation process that provides warnings for a number of releases prior to removal of a feature." That paragraph describes QEMU's scenario. NB, historically we've made arbitrary changes to the python code since it was not considered public API. If we make it official public API, then we would actually need to start following our deprecation process for the python code too. > With that in mind, I think it would be unwise for qemu.* not to do > the same; in particular, using a version number that's not <1.0.0 for > a package that is very much in flux will almost certainly break > people's expectations, and is also not something that you can easily > take back at a later time. I don't think it is that big a deal, and there is clear benefit to having the python code version match the QEMU version that it is the companioon to. Ultimately the versioning scheme just impacts on the version string conditionals people list for their dependancies. Apps consuming QEMU can handle any of the version schemes without much difference. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH v2 03/15] python: add VERSION file
On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: > Python infrastructure as it exists today is not capable reliably of > single-sourcing a package version from a parent directory. The authors > of pip are working to correct this, but as of today this is not possible > to my knowledge. > > The problem is that when using pip to build and install a python > package, it copies files over to a temporary directory and performs its > build there. This loses access to any information in the parent > directory, including git itself. > > Further, Python versions have a standard (PEP 440) that may or may not > follow QEMU's versioning. In general, it does; but naturally QEMU does > not follow PEP 440. To avoid any automatically-generated conflict, a > manual version file is preferred. > > > I am proposing: > > - Python core tooling synchronizes with the QEMU version directly > (5.2.0, 5.1.1, 5.3.0, etc.) > > - In the event that a Python package needs to be updated independently > of the QEMU version, a pre-release alpha version should be preferred, > but *only* after inclusion to the qemu development or stable branches. > > e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's > release. > > - The Python core tooling makes absolutely no version compatibility > checks or constraints. It *may* work with releases of QEMU from the > past or future, but it is not required to. > > i.e., "qemu.core" will always remain in lock-step with QEMU. > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp > and begin indepedently versioning such a package separately from the > QEMU version it accompanies. I think this need to be considered very carefully. I'm not overly familiar with the Python ecosystem but it would appear that, despite PEP 440 not mandating this, many (most?) of the packages uploaded to PyPi are using semantic versioning. With that in mind, I think it would be unwise for qemu.* not to do the same; in particular, using a version number that's not <1.0.0 for a package that is very much in flux will almost certainly break people's expectations, and is also not something that you can easily take back at a later time. -- Andrea Bolognani / Red Hat / Virtualization
Re: [PATCH v2 03/15] python: add VERSION file
On 10/14/20 10:29 AM, John Snow wrote: Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible to my knowledge. The problem is that when using pip to build and install a python package, it copies files over to a temporary directory and performs its build there. This loses access to any information in the parent directory, including git itself. See also: https://github.com/pypa/pip/issues/7549 https://github.com/pypa/pip/issues/7555 Further, Python versions have a standard (PEP 440) that may or may not follow QEMU's versioning. In general, it does; but naturally QEMU does not follow PEP 440. To avoid any automatically-generated conflict, a manual version file is preferred. I am proposing: - Python core tooling synchronizes with the QEMU version directly (5.2.0, 5.1.1, 5.3.0, etc.) - In the event that a Python package needs to be updated independently of the QEMU version, a pre-release alpha version should be preferred, but *only* after inclusion to the qemu development or stable branches. e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's release. - The Python core tooling makes absolutely no version compatibility checks or constraints. It *may* work with releases of QEMU from the past or future, but it is not required to. i.e., "qemu.core" will always remain in lock-step with QEMU. - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp and begin indepedently versioning such a package separately from the QEMU version it accompanies. Implement this versioning scheme by adding a VERSION file and setting it to 5.2.0a1. Signed-off-by: John Snow --- python/VERSION | 1 + python/setup.cfg | 1 + 2 files changed, 2 insertions(+) create mode 100644 python/VERSION diff --git a/python/VERSION b/python/VERSION new file mode 100644 index 00..2e81039c82 --- /dev/null +++ b/python/VERSION @@ -0,0 +1 @@ +5.2.0a1 diff --git a/python/setup.cfg b/python/setup.cfg index 12b99a796e..260f7f4e94 100755 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -1,5 +1,6 @@ [metadata] name = qemu +version = file:VERSION maintainer = QEMU Developer Team maintainer_email = qemu-devel@nongnu.org url = https://www.qemu.org/
