On Tue, May 07, 2019 at 03:13:45PM +0100, Daniel P. Berrangé wrote: > On Mon, May 06, 2019 at 06:38:17PM -0300, Eduardo Habkost wrote: > > test-qapi.py doesn't force a specific encoding for stderr or > > stdout, but the reference files used by check-qapi-schema are in > > UTF-8. This breaks check-qapi-schema under certain circumstances > > (e.g. if using the C locale and Python < 3.7). > > > > We need to make sure test-qapi.py always generate UTF-8 output > > somehow. On Python 3.7+ we can do it using > > `sys.stdout.reconfigure(...)`, but we need a solution that works > > with older Python versions. > > > > Instead of trying a hack like reopening sys.stdout and > > sys.stderr, we can just tell Python to use UTF-8 for I/O encoding > > when running test-qapi.py. Do it by setting PYTHONIOENCODING. > > > > Reported-by: Thomas Huth <th...@redhat.com> > > Tested-by: Thomas Huth <th...@redhat.com> > > Signed-off-by: Eduardo Habkost <ehabk...@redhat.com> > > --- > > tests/Makefile.include | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > index 7c8b9c84b2..af88ab6f8b 100644 > > --- a/tests/Makefile.include > > +++ b/tests/Makefile.include > > @@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: > > tests/qemu-iotests-quick.sh qemu-img$(EXESUF) > > .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) > > $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: > > $(SRC_PATH)/%.json > > $(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \ > > - $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ > > + PYTHONIOENCODING=utf-8 $(PYTHON) > > $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ > > I see PYTHONIOENCODING exists since 2.6 which is nice. > > How about we actually change $(PYTHON) so that it always includes > PYTHONIOENCODING=utf-8 ? > > That way we avoid continuing to play whack-a-mole with more utf-8 > bugs in future. > > It would also let us revert this: > > commit de685ae5e9a4b523513033bd6cadc8187a227170 > Author: Markus Armbruster <arm...@redhat.com> > Date: Mon Jun 18 19:59:57 2018 +0200 > > qapi: Open files with encoding='utf-8' > > which had to provide separate logic for py2 vs py3 :-(
Not every Python script in the QEMU tree is run by our makefiles and scripts using $(PYTHON). We need to ensure our scripts and modules won't break when run directly from the command line, too. Setting PYTHONIOENCODING everywhere would just hide these bugs from us. -- Eduardo