Add python3.6 to the fedora container image: we need it to run the
linters against that explicit version to make sure we don't break our
minimum version promise.
Add pipenv so that we can fetch precise versions of pip packages we need
to guarantee test reproducability.
Signed-off-by: John Snow
This is intended to be a manually run, non-CI script.
Use tox to test the linters against all python versions from 3.6 to
3.9. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:
> sudo dnf install python36 python37 python38 python39
Unlike the
Add "make venv" to create the pipenv-managed virtual environment that
contains our explicitly pinned dependencies.
Add "make check" to run the python linters [in the host execution
environment].
Add "make venv-check" which combines the above two: create/update the
venv, then run the linters in
isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.
isort < 5.1.1 has a bug where it does not handle comments near import
statements correctly.
Require 5.1.2 or greater.
isort can be run with 'isort -c qemu' from the
This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)
The qemu packages are
Signed-off-by: John Snow
Reviewed-by: Cleber Rosa
---
python/.isort.cfg | 7 ---
python/setup.cfg | 8
2 files changed, 8 insertions(+), 7 deletions(-)
delete mode 100644 python/.isort.cfg
diff --git a/python/.isort.cfg b/python/.isort.cfg
deleted file mode 100644
index
0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.
0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.
Now that we are
Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build.
Ignore miscellaneous cached python confetti (__pycache__, *.pyc,
.mypy_cache).
Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).
Signed-off-by: John Snow
---
Try using avocado to manage our various tests; even though right now
they're only invoking shell scripts and not really running any
python-native code.
Create tests/, and add shell scripts which call out to mypy, flake8,
pylint and isort to enforce the standards in this directory.
Add
setuptools doesn't have a formal understanding of development requires,
but it has an optional feataures section. Fine; add a "devel" feature
and add the requirements to it.
To avoid duplication, we can modify pipenv to install qemu[devel]
instead. This enables us to run invocations like "pip
mypy supports reading its configuration values from a central project
configuration file; do so.
Signed-off-by: John Snow
---
python/mypy.ini | 4
python/setup.cfg | 5 +
2 files changed, 5 insertions(+), 4 deletions(-)
delete mode 100644 python/mypy.ini
diff --git a/python/mypy.ini
Delete the empty settings now that it's sharing a home with settings for
other tools.
pylint can now be run from this folder as "pylint qemu".
Signed-off-by: John Snow
Reviewed-by: Cleber Rosa
Tested-by: Cleber Rosa
---
python/qemu/machine/pylintrc | 58
Pylint 2.5.x - 2.7.x have regressions that make import checking
inconsistent, see:
https://github.com/PyCQA/pylint/issues/3609
https://github.com/PyCQA/pylint/issues/3624
https://github.com/PyCQA/pylint/issues/3651
Pinning to 2.4.4 is worse, because it mandates versions of shared
dependencies
flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.
flake8 can be run from the python/ directory with no arguments.
Signed-off-by: John Snow
Reviewed-by:
Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source
Instruct flake8 to avoid certain well-known directories created by
python tooling that it ought not check.
Note that at-present, nothing actually creates a ".venv" directory; but
it is in such widespread usage as a de-facto location for a developer's
virtual environment that it should be excluded
Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:
A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.
Signed-off-by: John Snow
pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.
pipenv uses two files to do this:
(1) Pipfile, which is similar in purpose and scope to what setup.cfg
lists. It specifies the requisite
We are specifying >= pylint 2.7.x for several reasons:
1. For setup.cfg support, added in pylint 2.5.x
2. To specify a version that has incompatibly dropped
bad-whitespace checks (2.6.x)
3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9
Signed-off-by: John Snow
---
python/Pipfile
When creating a source distribution via 'python3 setup.py sdist', the
VERSION and PACKAGE.rst files aren't bundled by default. Create a
MANIFEST.in file that instructs the build tools to include these so that
installation from source dists won't fail.
(This invocation is required by 'tox', as
Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible.
The problem is that when using pip to build and install a python
package, it copies
mypy is kind of weird about how it handles imports. For legacy reasons,
it won't load PEP 420 namespaces, because of logic implemented prior to
that becoming a standard.
So, if you plan on using any, you have to pass
--namespace-packages. Alright, fine.
Signed-off-by: John Snow
Reviewed-by:
Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
this package is for and who to contact for more information. This
document will be used as the landing page for the package on PyPI.
I am not yet using a
move python/qemu/*.py to python/qemu/[machine, qmp]/*.py and update import
directives across the tree.
This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into
Create a space for miscellaneous things that don't belong strictly in
"qemu.machine" nor "qemu.qmp" packages.
Signed-off-by: John Snow
---
python/qemu/machine/__init__.py | 8
python/qemu/utils/__init__.py | 23 +++
python/qemu/{machine =>
Based-on: <20210512214642.2803189-1-js...@redhat.com>
CI: https://gitlab.com/jsnow/qemu/-/pipelines/302010131
GitLab: https://gitlab.com/jsnow/qemu/-/tree/python-package-mk3
MR: https://gitlab.com/jsnow/qemu/-/merge_requests/4
I invite you to leave review comments on my mock merge request on
One less file resource to manage, and it helps quiet some pylint >=
2.8.0 warnings about not using a with-context manager for the open call.
Signed-off-by: John Snow
---
python/qemu/machine.py | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/python/qemu/machine.py
In a few cases, we can't use 'with ...' because they belong to
long-running classes that need those items to stay open at the end of
the block. We're handling it, so tell pylint to shush.
Signed-off-by: John Snow
---
tests/qemu-iotests/iotests.py| 2 +-
tests/qemu-iotests/testrunner.py | 1
pylint 2.8.x adds warnings whenever we use Popen calls without using
'with', so it's desirable to convert synchronous calls to run()
invocations where applicable.
(Though, this trades one pylint warning for another due to a pylint bug,
which I've silenced with a pragma and a link to the bug.)
More on the lines of quieting pylint 2.8.x, though to make it obvious
that we definitely handle the cleanup here, I elected to bolster the
close() method here.
1. Check for the process having terminated early more rigorously by
checking poll() directly.
2. Change the prompt read into an
Shift the open() call later so that the pylint pragma applies *only* to
that one open() call. Add a note that suggests why this is safe: the
resource is unconditionally cleaned up in _post_shutdown().
_post_shutdown is called after failed launches (see launch()), and
unconditionally after every
More pylint 2.8.x warning hushing: use open's context manager where it's
applicable to do so to avoid a warning.
Signed-off-by: John Snow
---
tests/qemu-iotests/iotests.py | 13 ++---
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/tests/qemu-iotests/iotests.py
gitlab CI: https://gitlab.com/jsnow/qemu/-/pipelines/301924893
branch: https://gitlab.com/jsnow/qemu/-/commits/python-package-pre-cleanup
This series serves as a pre-requisite for packaging the python series
and getting the linters running via CI. The first patch fixes a linter
error we've had
Fixes pylint warnings.
Signed-off-by: John Snow
Reviewed-by: Cleber Rosa
---
python/qemu/console_socket.py | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py
index ac21130e446..87237bebef7 100644
---
use run() instead of Popen() -- to assert to pylint that we are not
forgetting to close a long-running program.
Signed-off-by: John Snow
---
python/qemu/machine.py | 15 +--
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/python/qemu/machine.py b/python/qemu/machine.py
We manage cleaning up this resource ourselves. Pylint should shush.
Signed-off-by: John Snow
---
python/qemu/console_socket.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/qemu/console_socket.py b/python/qemu/console_socket.py
index 87237bebef7..8c4ff598ad7 100644
---
We handle this resource rather meticulously in
shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in
_do_shutdown().
Quiet this pylint warning here.
Signed-off-by: John Snow
---
python/qemu/machine.py | 3 +++
1 file changed, 3 insertions(+)
diff --git
* Kevin Wolf (kw...@redhat.com) wrote:
> Commit f61fe11aa6f broke hmp_loadvm() by adding an incorrect negation
> when converting from 0/-errno return values to a bool value. The result
> is that loadvm resumes the VM now if it failed and keeps it stopped if
> it failed. Fix it to restore the old
On 12/05/21 17:44, Stefan Hajnoczi wrote:
If we follow this strategy basically any data structure used
by coroutines needs its own fine-grained lock (like Java's Object base
class which has its own lock).
Maybe not all, but only those that use CoQueue? Interestingly, I was a
bit less okay
297 so far does not check the named tests, which reside in the tests/
directory (i.e. full path tests/qemu-iotests/tests). Fix it.
Thanks to the previous two commits, all named tests pass its scrutiny,
so we do not have to add anything to SKIP_FILES.
Signed-off-by: Max Reitz
Reviewed-by:
pylint complains that discards1_sha256 and all_discards_sha256 are first
set in non-__init__ methods.
These variables are not really class-variables anyway, so let them
instead be returned by start_postcopy(), thus silencing pylint.
Suggested-by: Vladimir Sementsov-Ogievskiy
Signed-off-by: Max
169 and 199 have been renamed and moved to tests/ (commit a44be0334be:
"iotests: rename and move 169 and 199 tests"), so we can drop them from
the skip list.
Signed-off-by: Max Reitz
Reviewed-by: Willian Rampazzo
Reviewed-by: Vladimir Sementsov-Ogievskiy
Reviewed-by: Kevin Wolf
---
There are a couple of things pylint takes issue with:
- The "time" import is unused
- The import order (iotests should come last)
- get_bitmap_hash() doesn't use @self and so should be a function
- Semicolons at the end of some lines
- Parentheses after "if"
- Some lines are too long (80
v1: https://lists.nongnu.org/archive/html/qemu-block/2021-03/msg01471.html
Hi,
When reviewing Vladimir’s new addition to tests/, I noticed that 297 so
far does not cover named tests. That isn’t so good.
This series makes it cover them, and because tests/ is rather sparse at
this point, I
On 05.05.21 10:02, Vladimir Sementsov-Ogievskiy wrote:
Hi!
Kindly remind. Didn't you forget? I'm responsible for these two tests,
let me know if you don't have time, and I'll resend this series :)
Sorry, I forgot indeed... v2 coming right up. :)
Max
On 06.05.21 11:06, Vladimir Sementsov-Ogievskiy wrote:
Hi all!
v3:
01-04,06,07: add Max's r-b.
01: improve commit msg and comments
03: improve commit msg
05: add more comments and qemu/atomic.h include
08: new, replacement for v2:08,09
Vladimir Sementsov-Ogievskiy (8):
On 10.05.21 21:04, Emanuele Giuseppe Esposito wrote:
pylint 2.8 introduces consider-using-with error, suggesting
to use the 'with' block statement when possible.
Modify all subprocess.Popen call to use the 'with' statement,
except the one in __init__ of QemuIoInteractive class, since
it is
On Thu, May 06, 2021 at 12:06:13PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> v3:
> 01-04,06,07: add Max's r-b.
> 01: improve commit msg and comments
> 03: improve commit msg
> 05: add more comments and qemu/atomic.h include
> 08: new, replacement for v2:08,09
>
> Vladimir
On Fri, May 07, 2021 at 06:59:01PM +0200, Greg Kurz wrote:
> Now that virtio-scsi-pci and virtio-blk-pci map 1 virtqueue per vCPU,
> a serious slow down may be observed on setups with a big enough number
> of vCPUs.
>
> Exemple with a pseries guest on a bi-POWER9 socket system (128 HW threads):
>
On Tue, May 11, 2021 at 12:41:54PM +0200, Philippe Mathieu-Daudé wrote:
> Trivial patches to keep VirtIOFeature arrays read-only
> (better safe than sorry).
>
> Philippe Mathieu-Daudé (3):
> hw/virtio: Pass virtio_feature_get_config_size() a const argument
> virtio-blk: Constify VirtIOFeature
On Mon, May 10, 2021 at 10:59:39AM +0200, Emanuele Giuseppe Esposito wrote:
> Progressmeter is protected by the AioContext mutex, which
> is taken by the block jobs and their caller (like blockdev).
>
> We would like to remove the dependency of block layer code on the
> AioContext mutex, since
On Mon, May 10, 2021 at 10:59:40AM +0200, Emanuele Giuseppe Esposito wrote:
> co-shared-resource is currently not thread-safe, as also reported
> in co-shared-resource.h. Add a QemuMutex because co_try_get_from_shres
> can also be invoked from non-coroutine context.
>
> Signed-off-by: Emanuele
On Mon, May 10, 2021 at 10:59:41AM +0200, Emanuele Giuseppe Esposito wrote:
> Divide the fields in AioTaskPool in IN and Status, and
> introduce a CoQueue instead of .wait to take care of suspending
> and resuming the calling coroutine, and a lock to protect the
> busy_tasks counter accesses and
Am 11.05.2021 um 10:35 hat Daniel P. Berrangé geschrieben:
> On Tue, May 11, 2021 at 01:55:18PM +0800, Li Zhijian wrote:
> > Image below user cases:
> > case 1:
> > ```
> > $ qemu-img create -f raw source.raw 1G
> > $ qemu-img create -f qcow2 -F raw -b source.raw ./source.raw
> > qemu-img info
Am 11.05.2021 um 10:23 hat Stefan Hajnoczi geschrieben:
> On Mon, Mar 22, 2021 at 09:23:24AM +, Stefan Hajnoczi wrote:
> > These patches add a qtest for the vhost-user-blk server. CI found several
> > issues that caused these patches to be dropped from Michael Tsirkin and
> > Kevin
> > Wolf's
Am 11.05.2021 um 10:25 hat Stefan Hajnoczi geschrieben:
> On Wed, Mar 31, 2021 at 03:27:27PM +0100, Stefan Hajnoczi wrote:
> > The checks in vu_blk_sect_range_ok() assume VIRTIO_BLK_SECTOR_SIZE is
> > equal to BDRV_SECTOR_SIZE. This is true, but let's add a
> > QEMU_BUILD_BUG_ON() to make it
On Mon, May 10, 2021 at 10:59:35AM +0200, Emanuele Giuseppe Esposito wrote:
> This serie of patches bring thread safety to the smaller APIs used by
> block-copy, namely ratelimit, progressmeter, co-shared-resource
> and aiotask.
> The end goal is to reduce the usage of the global
> AioContexlock
Klaus Jensen writes:
> Hi all,
>
> I need some help with grok'ing qdev busses. Stefan, Michael - David
> suggested on IRC that I CC'ed you guys since you might have solved a
> similar issue with virtio devices. I've tried to study how that works,
> but I'm not exactly sure how to apply it to
On Wed, May 12, 2021 at 12:40:03PM +0300, Roman Kagan wrote:
> On Mon, Apr 19, 2021 at 10:34:49AM +0100, Daniel P. Berrangé wrote:
> > On Fri, Apr 16, 2021 at 11:08:59AM +0300, Vladimir Sementsov-Ogievskiy
> > wrote:
> > > Detecting monitor by current coroutine works bad when we are not in
> > >
On Mon, Apr 19, 2021 at 10:34:49AM +0100, Daniel P. Berrangé wrote:
> On Fri, Apr 16, 2021 at 11:08:59AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > Detecting monitor by current coroutine works bad when we are not in
> > coroutine context. And that's exactly so in nbd reconnect code, where
> >
On Fri, Apr 16, 2021 at 11:08:57AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> To be reused in the following patch.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy
> ---
> block/nbd.c | 99 ++---
> 1 file changed, 57 insertions(+), 42
On Wed, 12 May 2021 at 04:39, Philippe Mathieu-Daudé wrote:
> IIUC, while we can have unattached drives, we can't (by design) have
> qdev unattached to qbus.
You can (and we do), but it is a bit of a problem because a
device not attached to a qbus will not get automatically reset,
and so you
Fix the issue from this patch:
[PATCH] block: Flush all children in generic code
>From 883833e29cb800b4d92b5d4736252f4004885191
Quorum driver do not have the primary child.
It will caused guest block flush issue when use quorum and NBD.
The vm guest flushes failed，and then guest filesystem is
12.05.2021 10:06, Paolo Bonzini wrote:
On 16/04/21 10:09, Vladimir Sementsov-Ogievskiy wrote:
We already have two similar helpers for other state. Let's add another
one for convenience.
Signed-off-by: Vladimir Sementsov-Ogievskiy
The whole usage of load-acquire for the connected state is
12.05.2021 09:56, Paolo Bonzini wrote:
On 16/04/21 10:08, Vladimir Sementsov-Ogievskiy wrote:
With the following patch we want to call wake coroutine from thread.
And it doesn't work with aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for
On Tue, May 11, 2021 at 12:41:54PM +0200, Philippe Mathieu-Daudé wrote:
Trivial patches to keep VirtIOFeature arrays read-only
(better safe than sorry).
Philippe Mathieu-Daudé (3):
hw/virtio: Pass virtio_feature_get_config_size() a const argument
virtio-blk: Constify VirtIOFeature
11.05.2021 15:28, Paolo Bonzini wrote:
On 10/05/21 13:28, Vladimir Sementsov-Ogievskiy wrote:
Could we instead add a c file and add the structure private? Then we'll have
progress_new() and progress_free() APIs instead.
This way, it would be a lot simpler to control that nobady use
On 16/04/21 10:09, Vladimir Sementsov-Ogievskiy wrote:
We already have two similar helpers for other state. Let's add another
one for convenience.
Signed-off-by: Vladimir Sementsov-Ogievskiy
The whole usage of load-acquire for the connected state is completely
meaningless, but that's not
On 16/04/21 10:08, Vladimir Sementsov-Ogievskiy wrote:
With the following patch we want to call wake coroutine from thread.
And it doesn't work with aio_co_wake:
Assume we have no iothreads.
Assume we have a coroutine A, which waits in the yield point for
external aio_co_wake(), and no progress
On 16/04/21 10:08, Vladimir Sementsov-Ogievskiy wrote:
The series substitutes "[PATCH v2 00/10] block/nbd: move connection code to separate
file"
Supersedes: <20210408140827.332915-1-vsement...@virtuozzo.com>
so it's called v3
block/nbd.c is overcomplicated. These series is a big refactoring,
11.05.2021 13:45, Roman Kagan wrote:
On Fri, Apr 16, 2021 at 11:08:54AM +0300, Vladimir Sementsov-Ogievskiy wrote:
Add arguments and logic to support nbd negotiation in the same thread
after successful connection.
Signed-off-by: Vladimir Sementsov-Ogievskiy
---
include/block/nbd.h | 9
12.05.2021 00:08, Roman Kagan wrote:
On Fri, Apr 16, 2021 at 11:08:56AM +0300, Vladimir Sementsov-Ogievskiy wrote:
Now, when thread can do negotiation and retry, it may run relatively
long. We need a mechanism to stop it, when user is not interested in
result anymore. So, on
12.05.2021 00:19, Eric Blake wrote:
On 3/24/21 3:51 PM, Vladimir Sementsov-Ogievskiy wrote:
We are going to support 64 bit write-zeroes requests. Now update the
limit variable. It's absolutely safe. The variable is set in some
drivers, and used in bdrv_co_do_pwrite_zeroes().
Update also
73 matches
Mail list logo
