[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

We found an agreement with Nick :-) I applied Nick's PR. The issue is now fixed:

vstinner@apu$ ./python -X dev -c 'import sys; print(sys.warnoptions)'
['default']

This PR also fix bpo-20361: -b and -bb options now have the highest priority.

The documentation is maybe not perfect, but Nick stated that he will update the 
doc with the implementation of his PEP 565.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 747f48e2e92390c44c72f52a1239959601cde157 by Victor Stinner in 
branch 'master':
bpo-32230: Set sys.warnoptions with -X dev (#4820)
https://github.com/python/cpython/commit/747f48e2e92390c44c72f52a1239959601cde157


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +4710

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

"-W options > PYTHONWARNINGS > BytesWarning > -X dev > default filters"

Hum, sorry, according to bpo-20361 (and bpo-32231 which has been marked as a 
duplicate of bpo-20361), -b and -bb options must have the priority over 
-W/PYTHONWARNINGS. So it gives something closer to:

-b and -bb > -W > PYTHONWARNINGS > ignore::BytesWarning > -X dev > default 
filters

where ignore::BytesWarning is only not added if -b or -bb is used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

If we wanted to go that way, then I'd go back to my original code order for - 
putting the '-X dev' setting first in sys.warnoptions, so both PYTHONWARNINGS 
and -W options override it.

Then https://bugs.python.org/issue32231 would determine the relative precedence 
of the BytesWarning filter. (I still think the interaction between "-Wd" and 
"-bb" is weird, but we can argue about that on the other issue).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

I prefer to not talk about sys.warnoptions since, as you wrote, the order is 
reversed.

IMHO the priority should be (high priority > low priority):

  -W options > PYTHONWARNINGS > BytesWarning > -X dev > default filters

With:

* BytesWarning: "ignore" by default, "default" for -b, "error" for -bb.
* default filters: 4 filters added in release mode

[('ignore', None, , None, 0),
 ('ignore', None, , None, 0),
 ('ignore', None, , None, 0),
 ('ignore', None, , None, 0)]

  or 1 filter for pydebug build:

[('default', None, , None, 0)]


Currently, only "-W options > PYTHONWARNINGS" goes into sys.warnoptions, 
whereas "BytesWarning > -X dev > default filters" is added by the warnings 
module ("init_filters()").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

Right, I hit some of those corner cases myself before I realised that 
pymain_add_warnings_options was the right place to inject "default" into 
sys.warnoptions after your refactoring improvements. It provides a nice pivot 
point where the command line machinery can decide relative priority and turn 
things into a neatly ordered list.

That gives the following priority order in my PR:

- first we add the filters from PYTHONWARNINGS
- then we optionally add an extra "default" entry based on 
core_config->dev_mode (which may be from either PYTHONDEVMODE or from "-X dev")
- then we add the filters from "-W" command line options

The "last added, first applied" behaviour of "warnings.simplefilter" then means 
that later entries in that list take precedence over earlier entries.

If we go on to also implement https://bugs.python.org/issue32231, then we'd add 
a 4th step to inject a "default::BytesWarning" or "error::BytesWarning" filter 
at the end based on cmdline->bytes_warning.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

> It isn't good to have "-X dev" do magical things that can't already be 
> replicated with other options.

While it might be nice to be able to replicate -X dev "manually", I didn't try 
to implement that. The current implementation of this option is more complex 
than what I expected, especially for warnings.

> Writing that issue made me realise another quirk with `-X dev` though: as a 
> command line option, it should really take precedence over PYTHONWARNINGS, 
> while remain subordinate to other explicit -W options.

IMHO -X dev should change the default, but let the developer overrides options 
set by -X dev. For example, -X dev -bb works as expected: raise BytesWarning, 
and this is a test for that. There is also a test for -X dev -W error: -W error 
has the priority over -X dev, I did that on purpose.

When I started to implement -X dev, I didn't expect so many corner cases. The 
problem is that the code reading the "configuration" (command line arguments, 
environment variables, a few configuration files, etc.) was spreaded around 
CPython code. You know that very well, since you wrote the PEP 432 :-) Fix the 
implementation of -X dev was my first motivation to work on the implementation 
of the PEP 432: bpo-32030.

For PYTHONWARNINGS vs -X dev, I'm not sure. It's rare to use PYTHONWARNINGS. I 
put PYTHONWARNINGS and -W options at the same level. If someone uses 
PYTHONWARNINGS, warnings are likely already well understood. Overriding 
PYTHONWARNINGS with -X dev can be seen as a bug. Sometimes, you cannot set 
command line options, only environment variable. There is -X dev, but there is 
also PYTHONDEVMODE=1 ...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

In relation to the utility of sys.warnoptions: it also lets other modules (like 
the unittest module) inspect the warnings that have been specified on the 
command line.

If it wasn't for that use case, I wouldn't be concerned about the fact that `-X 
dev` doesn't currently set it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

No, we don't need to worry about BytesWarning if neither -b nor -bb are set, as 
the interpreter doesn't even emit the warning in that case.

The only thing the warnings machinery gets used for is to print the messages 
and/or raise the exceptions (so it would arguably be clearer to omit the filter 
entirely in the case where the flag isn't set at all).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

First I proposed that -X dev behaves as -Wd -b, but Antoine Pitrou and Serhiy 
Storchaka asked to remove -b:

https://mail.python.org/pipermail/python-dev/2017-November/150517.html
https://mail.python.org/pipermail/python-dev/2017-November/150519.html
https://mail.python.org/pipermail/python-dev/2017-November/150520.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

I filed https://bugs.python.org/issue32231 to cover having -bb always override 
other warnings settings when it comes to handling BytesWarning.

Writing that issue made me realise another quirk with `-X dev` though: as a 
command line option, it should really take precedence over PYTHONWARNINGS, 
while remain subordinate to other explicit -W options.

I've adjusted the PR to implement that change, but haven't added a test case 
that ensures the behaviour persists.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

> If we want "-bb" to take precedence over the passed in "-W" settings

It's not only that. Without -b nor -bb, the default::Warning filter should 
still come after ignore::BytesWarning, since these warnings must be ignored by 
default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

It isn't good to have "-X dev" do magical things that can't already be 
replicated with other options.

If we want "-bb" to take precedence over the passed in "-W" settings (which I 
agree would be a reasonable change), we should make that change universally, 
not restrict it to "-X dev".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Change by Nick Coghlan :


--
keywords: +patch
pull_requests: +4637
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

I was explicitly asked to not change the behaviour of BytesWarning with -X dev.

-X dev documentation doesn't menton -Wd/-Wdefault anymore:
https://docs.python.org/dev/using/cmdline.html#id5

"Warning filters: add a filter to display all warnings ("default" action), 
except of BytesWarning which still depends on the -b option, and use "always" 
action for ResourceWarning warnings. For example, display DeprecationWarning 
warnings."

Another implementation issue is subprocess._args_from_interpreter_flags() which 
was unable to properly "rebuild" the -X dev option from sys.warnoptions. I mean 
that it's now simpler and safer with sys.flags.dev_mode.

But I agree that the fact that -X dev doesn't touch sys.warnoptions is 
surprising and can cause issues. IMHO sys.warnoptions shouldn't exist in the 
first place. It looks like a hack to pass options from the C main() function to 
the warnings initilization code. But it's not like we can remove it right now 
:-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

Nick Coghlan  added the comment:

Note that this change will make "-X dev" effectively treat "-bb" and "-b" as 
equivalent, the same way "-Wd" already does. I figure that's OK, since it means 
the runtime behaviour will match the documentation (which says that "-X dev" 
implies "-Wd".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[STINNER Victor]

STINNER Victor  added the comment:

I did it on purpose. -Wdefault adds the filter at the start, whereas -Xdev
adds the filter at the end to respect BytesWarning. See tests in
test_cmd_line and bpo-32089.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32230] -X dev doesn't set sys.warnoptions

[Nick Coghlan]

New submission from Nick Coghlan :

The `-X dev` option is currently integrated directly with the warnings module, 
rather than working indirectly through `sys.warnoptions`.

This means that any third party code that currently checks sys.warnoptions will 
need to be updated to check sys.flags.dev_mode as well.

Rather than doing that, I propose that we instead change the way dev mode works 
to *literally* be equivalent to `-Wdefault`, and remove the direct integration 
with the warnings machinery.

(PR for that coming shortly)

--
assignee: ncoghlan
messages: 307709
nosy: ncoghlan, vstinner
priority: normal
severity: normal
stage: test needed
status: open
title: -X dev doesn't set sys.warnoptions
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com