Python logging (was Re: What should go to stdout/stderr and why Python logging write everything to stderr?)

[Weatherby,Gerard]

FWIW, it wasn’t too difficult to build a logging handler to send messages to 
Slack.

https://pypi.org/project/slack-webclient-logger/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-05 12:29:33 -0800, Grant Edwards wrote:
> On 2023-01-05, Thomas Passin  wrote:
> 
> > The logging system is so configurable that...
> 
> I find it almost impossible to use unless I copy a working example I
> find somewhere. ;)

I usually copy the working example from my previous project ;-).

I think the general structure is clear enough. You need a formatter to
format your log messages, a handler to actually write them and finally a
logger to determine what goes where. I can look up the details in the
docs but generally the logging is always the same (except for path names
and log levels), so I can just copy the config from last time and adjust
those.

So I might have a config.py like this:

# ...
logging = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s %(levelname)s %(name)s %(funcName)s 
%(lineno)d | %(message)s"
}
},
"handlers": {
"file": {
"class": "switchinglogfilehandlers.TimeoutSwitchingFileHandler",
"formatter": "standard",
"filename": "/var/log/www/XXX.hjp.at/XXX.",
},
},
"loggers": {
"": {
"handlers": ["file"],
"level": "INFO"
}
}
}
# ...

And then my application would start like this:

import logging
import logging.config
import config
logging.config.dictConfig(config.logging)
log = logging.getLogger(__name__)

Plus typically every other source file contains

import logging
log = logging.getLogger(__name__)

somewhere near the start.

Then I can just scatter my log.debug(...) or whatever whereever I want.
When I decide that I need debug output from one module, I'll just add a
logger. Or if some library is too chatty I'll add another logger to shut
it up - no programming, just a few extra lines in a dict.

(Instead of a config.py I might use a json file, especially if I expect
the config to change often.)

> I'm not at all surprised that the OP didn't understand how it
> works.

It probably helps to have worked with log4j before that. The structure
is very similar, although I find Python logging easier to use (but then
I never did much Java programming so that's probably just a matter of
familiarity.

hp

PS: The TimeoutSwitchingFileHandler mentioned above is one I wrote
myself. You can find it on PyPI, but be warned that the
documentation is (still) quite lacking.

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: asyncio and tkinter (was: What should go to stdout/stderr and why Python logging write everything to stderr?)

[Thomas Passin]

On 1/5/2023 7:52 PM, Stefan Ram wrote:

Thomas Passin  writes:

On 1/5/2023 4:24 PM, Stefan Ram wrote:

You often can replace threads in tkinter by coroutines using
asyncio when you write a replacement for the mainloop of
tkinter that uses asyncio. Now, try to read only the official
documentation of asyncio and tkinter and figure out only from
this how to get such a program to work!

Cool! Can we have a link, please?


   I do not post links, but tried to create a minimal example app.


Thanks!  That's not exactly obvious, is it?  Congratulations on getting 
it working.



import asyncio
import logging
import tkinter

# This program was tested on Windows, it is experimental.
# It should open a tkinter root window.
# Ctrl-E should create a task. You should be able to create
# a new task even while one task is already running.
# Each task will end after about 10 seconds.
# Ctrl-X should exit the process.
# Or, use the menu "Action".

# I (Stefan Ram) designed and wrote the program, but especially
# the code in "app_async_mainloop" and "terminate_tasks" was
# written following educational material from the World-Wide Web.

class app_class( tkinter.Tk ):
 def __init__( self, *args, **kwargs ):
 self.is_running = 1_000_000
 super().__init__( *args, **kwargs )
 root = self
 root.protocol( "WM_DELETE_WINDOW", self.app_exit_macro )
 async def app_example_task( self, example_label ):
 try:
 for i in range( 10 ):
 example_label[ 'text' ]=str( i )
 await asyncio.sleep( 1 )
 except asyncio.CancelledError:
 pass
 example_label.destroy()
 def app_example_macro( self ):
 root = self
 example_label=tkinter.Label( root )
 example_label.pack()
 asyncio.get_running_loop().create_task\
 ( self.app_example_task( example_label ))
 root = self
 def terminate_tasks( self ):
 loop = asyncio.get_running_loop()
 pending = asyncio.all_tasks( loop=loop )
 label_tasks = []
 for task in pending:
 if 'app_example_task' in str( task ):
 label_tasks.append( task )
 task.cancel()
 group = asyncio.gather( *label_tasks, return_exceptions=True )
 try:
 loop.run_until_complete( group )
 except AssertionError:
 # ignoring an assertion error on Windows I do not understand
 pass
 def app_exit_macro( self ):
 self.terminate_tasks()
 self.is_running = 99
 root = self
 root.destroy()
 def app_add_basemenu( self, example=False ):
 root = self
 menubar = tkinter.Menu( root )
 menu = tkinter.Menu( menubar, tearoff=0 )
 name = "Actions"; menu = tkinter.Menu( menubar, tearoff=0 )
 if example:
 text = "Example";
 callback = self.app_example_macro; menu.add_command\
 ( label=text, underline=0, command=callback,
   accelerator="Control-e" );
 root.bind\
 ( "", lambda event, callback=callback:callback() )
 text = "Exit";
 callback = self.app_exit_macro
 menu.add_command\
 ( label=text, underline=1,
   command=callback, accelerator="Control-x" )
 root.bind\
 ( "", lambda event,callback=callback: callback() )
 menubar.add_cascade( label=name, underline=0, menu=menu )
 root.config( menu=menubar )
 async def app_async_mainloop( self ):
 root = self
 root.willdispatch()
 root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
 while self.is_running > 0:
 if self.is_running < 1_000_000: self.is_running -= 1
 try:
 await asyncio.sleep\
 ( tkinter._tkinter.getbusywaitinterval() / 10_000 )
 root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
 root.update()
 except tkinter.TclError:
 break
 async def app_async_main( self ):
 try:
 await self.app_async_mainloop()
 except asyncio.CancelledError:
 logging.debug( f'asyncio.CancelledError' )
 def app_async_run( self ):
 asyncio.run( self.app_async_main() )

app = app_class()
app.app_add_basemenu(example=True)
app.app_async_run()





--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/5/2023 4:24 PM, Stefan Ram wrote:

You often can replace threads in tkinter by coroutines using
   asyncio when you write a replacement for the mainloop of
   tkinter that uses asyncio. Now, try to read only the official
   documentation of asyncio and tkinter and figure out only from
   this how to get such a program to work!


Cool! Can we have a link, please?

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/5/2023 4:28 PM, Weatherby,Gerard wrote:

logging.basicConfig()
logging.info(“Nice to know”)
logging.debug(“Details for when things are funky”)
logging.warn(“Trouble is brewing”)


Not quite -

>>> import logging
>>> logging.basicConfig()
>>> logging.info("Nice to know")
>>> logging.debug("Details for when things are funky")
>>> logging.warn("Trouble is brewing")
:1: DeprecationWarning: The 'warn' function is deprecated, use 
'warning' instead

WARNING:root:Trouble is brewing

Only warning level messages are displayed, unless you do some 
configuration, as mentioned in the logging Howto page:


"The default level is WARNING, which means that only events of this 
level and above will be tracked, unless the logging package is 
configured to do otherwise."


So following the Howto:
>>> logging.basicConfig(level=logging.DEBUG)
>>> logging.info('So should this')
INFO:root:So should this

: Finally!

Not quite straightforward, though it is in the Howto.  Now about those 
handlers ...



From: Python-list  on behalf of 
Grant Edwards 
Date: Thursday, January 5, 2023 at 3:31 PM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 2023-01-05, Thomas Passin  wrote:


The logging system is so configurable that...


I find it almost impossible to use unless I copy a working example I
find somewhere. ;)

I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.

--
Grant




--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$>


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-05, Weatherby,Gerard  wrote:
> logging.basicConfig()
> logging.info(“Nice to know”)
> logging.debug(“Details for when things are funky”)
> logging.warn(“Trouble is brewing”)

I always seem to need something slightly more complex. Usually something like:

 * Specify a log level on the command line.
 * Allow logging to a file as specified on the command line.
 * Optional timestamps for log messages.

I know, without doubt, that it can easily do all those things. I just
never seem to be able to figure out how unless I can find example code
to look at.

--
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

logging.basicConfig()
logging.info(“Nice to know”)
logging.debug(“Details for when things are funky”)
logging.warn(“Trouble is brewing”)

From: Python-list  on 
behalf of Grant Edwards 
Date: Thursday, January 5, 2023 at 3:31 PM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 2023-01-05, Thomas Passin  wrote:

> The logging system is so configurable that...

I find it almost impossible to use unless I copy a working example I
find somewhere. ;)

I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.

--
Grant




--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/5/2023 3:29 PM, Grant Edwards wrote:

On 2023-01-05, Thomas Passin  wrote:


The logging system is so configurable that...


I find it almost impossible to use unless I copy a working example I
find somewhere. ;)

I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.


I know what you mean!  I have never used the Python logging package, but 
I inherited a java project that uses Log4j.  Talk about obscure!  The 
python docs are a wonder of clarity beside Log4j.  So most of my Log4j 
configuration has been by copy-pasta, with occasional surprises and 
Log4j bug bites.  Like a handler that deleted its own previous log file 
when it rolled over at midnight - a "feature" that wasn't advertised.


I will say, though, that having used a bit of Log4j must have been 
helpful, since much of what is in the Python logging package docs seemed 
somewhat familiar and even seemed to make sense.


BTW, I was able to re-create that java project using Jython for about 
90% of it.  Way better!


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-05, Thomas Passin  wrote:

> The logging system is so configurable that...

I find it almost impossible to use unless I copy a working example I
find somewhere. ;)

I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.

--
Grant




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/5/2023 2:18 PM, Peter J. Holzer wrote:

On 2023-01-05 08:31:40 -0500, Thomas Passin wrote:

The logging system is so configurable that a user could set a different
destination for each level of logging.  So it seems that the O.P.'s original
question about why the package's developers choose stderr for all levels can
be answered: "They didn't".


Which is almost exactly what I wrote in
<20230104162154.uzljittbs6xwt...@hjp.at> ;-)


Yup, two minds are better than one :)

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-05 08:31:40 -0500, Thomas Passin wrote:
> The logging system is so configurable that a user could set a different
> destination for each level of logging.  So it seems that the O.P.'s original
> question about why the package's developers choose stderr for all levels can
> be answered: "They didn't".

Which is almost exactly what I wrote in
<20230104162154.uzljittbs6xwt...@hjp.at> ;-)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/5/2023 6:27 AM, Peter J. Holzer wrote:

On 2023-01-04 12:32:40 -0500, Thomas Passin wrote:

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:

The logging module write everything to stderr no matter which logging
level is used.


The OP wrote this, but it's not so by default.


By default it is - sort of.

That is all log messages go to stderr, but not all log levels are
logged.


Actually, I misread something else that made me think that non-warning 
messages would go to stdout, but I was mistaken.  As it happens, the 
page in the Python docs only mention stderr twice, both in the same 
paragraph:


"logging.lastResort
A “handler of last resort” is available through this attribute. This is 
a StreamHandler writing to sys.stderr with a level of WARNING, and is 
used to handle logging events in the absence of any logging 
configuration.  The end result is to just print the message to sys.stderr."


Nowhere else on this page is the printed output destination mentioned, 
except that it gets printed on the console, which could be either stderr 
or stdout.


The logging system is so configurable that a user could set a different 
destination for each level of logging.  So it seems that the O.P.'s 
original question about why the package's developers choose stderr for 
all levels can be answered: "They didn't".


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-04 12:32:40 -0500, Thomas Passin wrote:
> On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:
> > The logging module write everything to stderr no matter which logging
> > level is used.
> 
> The OP wrote this, but it's not so by default.

By default it is - sort of. 

That is all log messages go to stderr, but not all log levels are
logged.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Thu, 5 Jan 2023 at 05:06, Barry Scott  wrote:
>
>
> On 03/01/2023 21:24, c.bu...@posteo.jp wrote:
> > Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> >> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
> >
> > But I don't want to make all log levels go to stdout. Just DEBUG and
> > INFO. But this would be a workaround.
> >
> > The main question here is why does Python deciecded to make all logs
> > go to stderr?
> > Maybe I totally misunderstood the intention of logging.info()?! Isn't
> > this the "usual applicaton output"?
> >
> > If not, what messages should go into logging.info()? Can you name me
> > some examples?
>
> Example:
>
> write an app that prints the contents of a file.
>
> The application output is the contents of the file.
>
> The logging might be this when all is working:
>
> INFO About to open 
> INFO Wrote  bytes from 
>
> The logging might be this when there is a problem:
>
> INFO About to open 
> ERROR Failed to open  - 
>
> Does that help?

Notably, the info lines can provide context for an error. For example:

INFO: Loading module XYZ
WARN: Unrecognized module option frobnicate=1
INFO: Using default spamination of 4
ERROR: Insufficient ham to spaminate

This tells a hypothetical (but only slightly hypothetical) story of a
module that was given an incorrect or misspelled option, and which
then failed to load as a consequence. Without the INFO lines, it would
be harder to figure out what the problem was, but they are still part
of logging, not output, and belong in the same place as the warnings
and errors.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


Example:

write an app that prints the contents of a file.

The application output is the contents of the file.

The logging might be this when all is working:

INFO About to open 
INFO Wrote  bytes from 

The logging might be this when there is a problem:

INFO About to open 
ERROR Failed to open  - 

Does that help?

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


It is up to you, the designer of an application, to decide how it works.
You will take into account conventions that your users will expect you 
to follow.


If the logging module helps you then use it, but you are not forced by 
logging to
design your app is a particular way. The default behavior of the logging 
module is
a generally useful default, but its only a default. You are free to 
setup logging to

meet your needs.

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:

The logging module write everything to stderr no matter which logging
level is used.


The OP wrote this, but it's not so by default.  There is a HOW-TO page 
that explains this and how to get the logging package to log everything 
to a file, along with many other useful bits of information -


https://docs.python.org/3/howto/logging.html

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-03 21:24:20 +, c.bu...@posteo.jp wrote:
> The main question here is why does Python deciecded to make all logs go to
> stderr?

It doesn't.

The Python logging system provides a plethora of handlers to log to lots
of different places. Only the StreamHandler defaults to writing to
stderr, and even that can log to any stream - just pass it as an
argument.

If none of existing handlers does what you want you can always write
your own.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-04 16:46:37 +1100, Chris Angelico wrote:
> On Wed, 4 Jan 2023 at 15:11, Eryk Sun  wrote:
> > On 1/3/23, Chris Angelico  wrote:
> > > writing the FD is the same as using stdout
> >
> > Except stdout may be buffered.

Maybe I'm overly lax in my terminology, but I frequently use the term
"stdout" as a shorthand for "standard output" which refers to file
descriptor 1, not just to the C FILE* of that name. Or Python's
sys.stdout which is a different thing again (but also is a frontend to
FD 1).

> > One should probably flush the buffer
> > before each raw write to the file descriptor.
> 
> FDs can also be buffered.

That depends on what you mean by "buffered". A write to an fd referring
to a file on a disk will not usually affect the disk, true. But it will
be immediately visible to other processes reading that file.

A write to a terminal attached via a serial cable will also be written
to a buffer first (in the OS or maybe inside the UART), but it will
be sent to the terminal as fast as the hardware allows.

So. sure, the writes are "buffered", but is that meaningful in this
context?

Reading from a terminal also often involves a buffer, but in a very
different sense: Here the OS buffers each line to implement line editing
functions. If the kernel didn't do that each application which could
possibly read from a terminal would have to implement that itself. and
it was decided back in the 1970's that this wasn't a good use of
resources. (Of course these days many programs which didn't do that
originally (like shells) do implement line editing (often via
readline).)

> If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.

I don't think fsync() will have an effect on an FD connected to a
terminal (or a pipe or a socket).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 06:46, Chris Angelico wrote:

I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1"


C++ has this feature:

Quote from https://en.cppreference.com/w/cpp/io/cin

"Once |std::cin| is constructed, std::cin.tie() returns ::cout 
, and likewise, 
std::wcin.tie() returns ::wcout 
. This means that any 
formatted input operation on |std::cin| forces a call to std::cout 
.flush() if any characters are 
pending for output."


Barry

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 02:26, Chris Angelico wrote:

Reading/writing the FD is the same as using stdout (technically you'd
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.


I think the logic is more like checking that stdin is a tty then using
the tty it to read and write the tty. This works with stdout redirected.
Also stdin is likely only open with read access.

For example the way to prevent ssh-add from prompting in the terminal is 
this:


$ ssh-add ~/.ssh/id_rsa https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[2QdxY4RzWzUUiLuE]

On 2023-01-04 at 12:02:55 +,
"Weatherby,Gerard"  wrote:

> Dealing with stdout / stderr is bash is just syntax. I don’t always
> remember it off the top of my head but … stackoverflow.com.

https://tldp.org/LDP/abs/html/io-redirection.html

> On Linux at least it’s possible to pipe to arbitrary streams, it
> seems. The following uses bash to write “Hi” to the file “third”
> opened by Python. I determined the file was 5 empirically.

Empirically?

> import os
> import subprocess
> command= 'echo Hi >/dev/fd/5'
> fd = os.open("third",os.O_WRONLY|os.O_CREAT)

command = f"echo Hi >/dev/fd/{fd}"

Or:

command = f"echo Hi >&{fd}"

> os.set_inheritable(fd,True)
> 
> subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)

By the time I'm that far down that path, I usually write a separate
function to call fork, open, close, and exec myself (old habits die
hard).  In the end, there's no reward for writing and maintaining shell
programs in Python.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

Dealing with stdout / stderr is bash is just syntax. I don’t always remember it 
off the top of my head but … stackoverflow.com.

On Linux at least it’s possible to pipe to arbitrary streams, it seems. The 
following uses bash to write “Hi” to the file “third” opened by Python. I 
determined the file was 5 empirically.


import os
import subprocess
command= 'echo Hi >/dev/fd/5'
fd = os.open("third",os.O_WRONLY|os.O_CREAT)
os.set_inheritable(fd,True)

subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)

From: Python-list  on 
behalf of Michael Torrie 
Date: Tuesday, January 3, 2023 at 5:18 PM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***


Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams.  Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.




--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 17:26, Eryk Sun  wrote:
>
> On 1/3/23, Chris Angelico  wrote:
> >
> > FDs can also be buffered. If it's buffering you want to avoid, don't
> > mess around with exactly which one you're writing to, just flush.
>
> I meant to flush a C FILE stream or Python file before writing
> directly to the file descriptor, in order to avoid out-of-sequence and
> interlaced writes that make no sense. Any OS buffering on the file
> doesn't matter in this regard.

True, that can help. If you're about to prompt for a password, though,
OS buffering can most certainly make a difference. And it doesn't hurt
to explicitly flush when you need it flushed.

I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1", but that's not going to work when you use certain
keyboard-input routines (ISTR readline was one of the few things that
*didn't* have that problem, but I may be misremembering), so the
general strategy was: print your prompt, flush stdout, THEN request
input.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Chris Angelico  wrote:
>
> FDs can also be buffered. If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.

I meant to flush a C FILE stream or Python file before writing
directly to the file descriptor, in order to avoid out-of-sequence and
interlaced writes that make no sense. Any OS buffering on the file
doesn't matter in this regard.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Grant Edwards  wrote:
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?

POSIX has required "/dev/tty" for 20 years, or longer. It's in the
2004 edition of the spec, which was a minor update to the 2001
edition.

https://pubs.opengroup.org/onlinepubs/95399/basedefs/xbd_chap10.html

> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...

FYI, that wouldn't work on Windows. The "\\.\CONIN$" and "\\.\CONOUT$"
files can be opened with read-write access, but it's for a different
purpose.

A console input buffer can be read from via read() or WinAPI
ReadFile(). It can also be read from using IOCTLs that work with
16-bit wide-character strings, which is the basis of WinAPI
ReadConsoleW() and ReadConsoleInputW(). A console input buffer can be
*written to* via the IOCTL-based function WriteConsoleInputW().

A console screen buffer can be written to via write() or WinAPI
WriteFile(). It can also be written to using IOCTLs that work with
16-bit wide-character strings, which is the basis of WriteConsoleW()
and WriteConsoleOutputW(). A console screen buffer can be *read from*
via the IOCTL-based function ReadConsoleOutputW().
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 16:21, Grant Edwards  wrote:
>
> On 2023-01-04, Chris Angelico  wrote:
> > On Wed, 4 Jan 2023 at 09:52, Grant Edwards  
> > wrote:
> >>
> >>> I can't think of a specific example, but I know I have piped the output
> >>> of a program while at the same time interacting with a prompt on stderr.
> >>> A rare thing, though.
> >>
> >> Programs that ask for passwords often do it by reading/writing to
> >> fd 0 or /dev/tty so that stdout is unaffected.
> >
> > Reading/writing the FD is the same as using stdout.
>
> No, stdout is fd 1.
>
> > (technically you'd write to fd 1 and read from fd 0),

I assumed that you were using a shorthand. My bad.

> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...

Writing to FD 0 assumes that (a) it was originally connected to the
TTY, and (b) it hasn't been redirected. Not a reliable assumption.

rosuav@sikorsky:~/tmp$ cat testfds.py
import os
for fd in range(3):
os.write(fd, b"This is on FD %d\n" % fd)
rosuav@sikorsky:~/tmp$ python3 testfds.py
This is on FD 0
This is on FD 1
This is on FD 2
rosuav@sikorsky:~/tmp$ echo | python3 testfds.py
Traceback (most recent call last):
  File "/home/rosuav/tmp/testfds.py", line 3, in 
os.write(fd, b"This is on FD %d\n" % fd)
OSError: [Errno 9] Bad file descriptor

Writing to fd 0 might work if stdout is redirected, but it won't work
if stdin is redirected, so honestly, I don't think this hack is at all
reliable.

> > but yes, can use /dev/tty to reach for the console directly.
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?

No idea. Probably. It's definitely possible that someone tries it with
stdout redirected and goes "oh hey, writing to fd 0 still works", and
never tries it with stdin redirected. It's very common for programmers
to see that something works without knowing WHY it works :)

By the way, I can reinstate the behaviour of "FD 0 writes to the TTY"
in bash by redirecting it:

rosuav@sikorsky:~/tmp$ python3 testfds.py >/dev/null 0<>/dev/tty
This is on FD 0
This is on FD 2

which is another neat demonstration that bash has more power than you
will ever need...

(though redirecting FDs above 2 in read/write mode can be *incredibly* useful.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 15:11, Eryk Sun  wrote:
>
> On 1/3/23, Chris Angelico  wrote:
> >
> > writing the FD is the same as using stdout
>
> Except stdout may be buffered. One should probably flush the buffer
> before each raw write to the file descriptor.

FDs can also be buffered. If it's buffering you want to avoid, don't
mess around with exactly which one you're writing to, just flush.

> > use /dev/tty to reach for the console directly.
>
> On Windows, open "CON" (read or write), "CONIN$" (read-write), or
> "CONOUT$" (read-write). Or use the more explicit device paths
> "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
> console, one can call WinAPI AllocConsole() or
> AttachConsole(process_id) to obtain one.

Yeah, /dev/tty is the Unix option, Windows has its own way, but
whatever platform you're on, there's a way to directly reach the
console.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-04, Chris Angelico  wrote:
> On Wed, 4 Jan 2023 at 09:52, Grant Edwards  wrote:
>>
>>> I can't think of a specific example, but I know I have piped the output
>>> of a program while at the same time interacting with a prompt on stderr.
>>> A rare thing, though.
>>
>> Programs that ask for passwords often do it by reading/writing to
>> fd 0 or /dev/tty so that stdout is unaffected.
>
> Reading/writing the FD is the same as using stdout.

No, stdout is fd 1.

> (technically you'd write to fd 1 and read from fd 0),

No, the whole point is _not_ to write to stdout (which would corrupt
the program's output). The way I remember it was that you wrote the
prompt to fd 0, and then read the password from fd 0. That way it
worked even when fd 1 (stdout) was redirected. It still failed if
stdin was redirected...

IIRC, you can't usually write to the stdin stream in C (or Python), so
you had to write(0,promptstr,strlen(promptstr)) instead of
printf("%s",promptstr) or fputs(promptstr,stdin).

> but yes, can use /dev/tty to reach for the console directly.

That's definitely a better option, but I'm pretty sure I've seen
multiple examples over the decades where fd 0 was used instead. Was
/dev/tty a "recent" enough invention that I would have seen
application code that was written before it existed? Or maybe I was
just looking at code written by people who weren't aware of /dev/tty?

--
Grant
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Chris Angelico  wrote:
>
> writing the FD is the same as using stdout

Except stdout may be buffered. One should probably flush the buffer
before each raw write to the file descriptor.

> use /dev/tty to reach for the console directly.

On Windows, open "CON" (read or write), "CONIN$" (read-write), or
"CONOUT$" (read-write). Or use the more explicit device paths
"\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
console, one can call WinAPI AllocConsole() or
AttachConsole(process_id) to obtain one.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 10:28, Stefan Ram  wrote:
>
> c.bu...@posteo.jp writes:
> >But I don't want to make all log levels go to stdout. Just DEBUG and
> >INFO.
>
>   The following was stripped down from something some guy
>   posted on a web site, maybe it was "rkachach".

Yet another shining example of bad code that does what someone asks,
but will make life harder for the next person to come along and try to
work with that program. It's like if someone asked for a way to make
True equal to -1 instead of 1, for compatibility with BASIC. Bad idea.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 09:52, Grant Edwards  wrote:
>
> On 2023-01-03, Michael Torrie  wrote:
> > On 1/3/23 11:45, Keith Thompson wrote:
> >> MRAB  writes:
> >> [...]
> >>> The purpose of stderr is to display status messages, logging and error
> >>> messages, even user prompts, and not mess up the program's actual
> >>> output. This is important on a *nix system where you might be piping
> >>> the output of one program into the input of another.
> >>
> >> I would expect user prompts to be written to stdout, or perhaps to some
> >> system-specific stream like the current tty, not to stderr.  If a
> >> program has user prompts, it probably doesn't make sense to pipe its
> >> output to the input of another.
> >
> > I can't think of a specific example, but I know I have piped the output
> > of a program while at the same time interacting with a prompt on stderr.
> > A rare thing, though.
>
> Programs that ask for passwords often do it by reading/writing to fd 0
> or /dev/tty so that stdout is unaffected.
>

Reading/writing the FD is the same as using stdout (technically you'd
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Grant Edwards]

On 2023-01-03, Michael Torrie  wrote:
> On 1/3/23 11:45, Keith Thompson wrote:
>> MRAB  writes:
>> [...]
>>> The purpose of stderr is to display status messages, logging and error
>>> messages, even user prompts, and not mess up the program's actual 
>>> output. This is important on a *nix system where you might be piping
>>> the output of one program into the input of another.
>> 
>> I would expect user prompts to be written to stdout, or perhaps to some
>> system-specific stream like the current tty, not to stderr.  If a
>> program has user prompts, it probably doesn't make sense to pipe its
>> output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.

Programs that ask for passwords often do it by reading/writing to fd 0
or /dev/tty so that stdout is unaffected.

--
Grant

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 09:17, Michael Torrie  wrote:
>
> On 1/3/23 11:45, Keith Thompson wrote:
> > MRAB  writes:
> > [...]
> >> The purpose of stderr is to display status messages, logging and error
> >> messages, even user prompts, and not mess up the program's actual
> >> output. This is important on a *nix system where you might be piping
> >> the output of one program into the input of another.
> >
> > I would expect user prompts to be written to stdout, or perhaps to some
> > system-specific stream like the current tty, not to stderr.  If a
> > program has user prompts, it probably doesn't make sense to pipe its
> > output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
>  A rare thing, though.

That can certainly happen, but you may more often see things that go
directly to the TTY rather than using the standard streams at all. For
example, ssh and sudo will prompt on the terminal if they need a
password, to avoid interfering with the standard streams (which can
and often will be redirected).

> Maybe some day an interface and shell syntax will be developed to
> interact with an arbitrary number of standard streams.  Current piping
> syntax really only works well with one stream and even trying to use
> stderr and stdout with pipes and filters in a shell is awkward.

Yeah, maybe some day. And then maybe someone will use a time machine
to backport it into bash. You have at least ten available, with more
being possible but potentially causing conflicts:

https://www.gnu.org/software/bash/manual/html_node/Redirections.html

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[2QdxY4RzWzUUiLuE]

On 2023-01-03 at 21:24:20 +,
c.bu...@posteo.jp wrote:

> The main question here is why does Python deciecded to make all logs
> go to stderr?

It makes sense to send all logging to one destination, and stderr is
that place.

> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?

Aha.  This seems to be the misconception.  "Usual" in the sense of "the
application is running as usual," but not in the sense of "the output
the users usually see."

Logging is the process where applications write details that ordinarily
aren't visible to the user.  Logs are for forensics, after the fact.
There is an entire universe of software for managing (writing, reading,
archiving, retrieving, searching, etc.) log files.

At my last job, we wrote gigabytes of compressed logs every hour, and
read roughly two or three out of a million entries to track down
questions from our users.  Users wouldn't have understood 99% of what
was in the logs anyway.

I also worked a lot in embedded systems, like telephone switches.  Users
(i.e., the people with the telephones) didn't have access to the
switches, but when things went wrong, I could look at the switches' logs
to get the their view of what happened.

> If not, what messages should go into logging.info()? Can you name me some
> examples?

INFO level messages usually track "regular" happy-path progress through
your application:  reading config files, calculating results, writing
report.  Again, not meant for users (who would have their own progress
meters, textual or graphical), but for tracking down problems when there
are problems.  For one example, the name of the config file, even if you
only tell the users that you found a config file.

HTH
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Michael Torrie]

On 1/3/23 11:45, Keith Thompson wrote:
> MRAB  writes:
> [...]
>> The purpose of stderr is to display status messages, logging and error
>> messages, even user prompts, and not mess up the program's actual 
>> output. This is important on a *nix system where you might be piping
>> the output of one program into the input of another.
> 
> I would expect user prompts to be written to stdout, or perhaps to some
> system-specific stream like the current tty, not to stderr.  If a
> program has user prompts, it probably doesn't make sense to pipe its
> output to the input of another.

I can't think of a specific example, but I know I have piped the output
of a program while at the same time interacting with a prompt on stderr.
 A rare thing, though.

Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams.  Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Wed, 4 Jan 2023 at 08:25,  wrote:
>
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.

But why are DEBUG logs part of stdout? That doesn't make any sense.

> The main question here is why does Python deciecded to make all logs go
> to stderr?
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
>
> If not, what messages should go into logging.info()? Can you name me
> some examples?

Reports of what the program is doing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Cameron Simpson]

On Jan 3, 2023, at 10:38 AM, c.bu...@posteo.jp wrote:
this posting isn't about asking for a technical solution. My 
intention is to understand the design decision Python's core developers made in

context of that topic.

The logging module write everything to stderr no matter which logging
level is used.


Sure. They're logs.


The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.


Error messages to stderr. _Requested output_ to stdout. Good with me.


Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.


I entirely disagree.

Consider a shell pipeline:

cmd1 | cmd2 | cmd3

where `cmd2` and `cmd3` parse data coming down the pipes. Letting 
logging info into those data streams would produce garbage. This 
separation between core programme output and error/logging output is the 
basic reason that the two streams exist, and the reason logging goes to 
stderr by default.


You're conflating "normal output" with INFO, which is not a substitute 
for eg `print()`.



Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.


Basicly, logging data is not the normal output. So it does not get sent 
to stdout. The log level doesn't change that characteristic.



My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


I think you should, or you should fiddle the handlers as you previous 
resist doing if it is your goal.


But keep in mind that if you do that globally (eg in the root logger) it 
will affect _all_ logging calls, not merely your own, and other 
libraries will assume they can log at whatever level and not pollute 
stdout with their logs.


Basicly, logging isn't "output".

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[c . buhtz]

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs go 
to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, Weatherby,Gerard  wrote:
> If sys.stdout is a tty, it typically flushes on newline. e. g.

Sorry if I wasn't clear. Yes, a tty file will be buffered, but it's
line buffering, which isn't an issue as long as lines are written to
stdout. I was referring to full buffering of pipe and disk files. This
can be a problem when printing diagnostics. We went the information
written to the file immediately, not whenever the buffer happens to
fill up. Thus sys.stderr is unbuffered.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

If sys.stdout is a tty, it typically flushes on newline. e. g.

!/usr/bin/env python3
import time
import sys
print("No flush",end='',file=sys.stdout)
time.sleep(2)
print("flushed",file=sys.stdout)
time.sleep(5)

will print the “flushed” 5 seconds before the script ends

From: Python-list  on 
behalf of Eryk Sun 
Date: Tuesday, January 3, 2023 at 1:33 PM
To: c.bu...@posteo.jp 
Cc: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 1/3/23, c.bu...@posteo.jp  wrote:
>
> If the user request the usage info via "-h" it goes to stdout.

The standard file for application output is sys.stdout. Note that
sys.stdout may be buffered, particularly if it isn't a tty. When a
file is buffered, writes are aggregated and only written to the OS
file when the buffer fills up or is manually flushed.

> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.

The standard file for error messages and other diagnostic information
is sys.stderr. This file should never be buffered.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 11:51 AM, Stefan Ram wrote:

Thomas Passin  writes:

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:
Also, I think it would be confusing to sometimes have logging output go
to stdout and other times go to stderr.


   In UNIX, the output of a program can be redirected,
   so error messages written to the output might get appended
   to some result file and might not be seen by the user.


Yes, and in Windows, too.  Note that stderr messages can be written or 
redirected to a file, too, and then would not be seen on the terminal.



   Therefore, UNIX and C provide the unbuffered "stderr" stream
   that can be redirected separately.

   The following example shows how logging might write to stdout.

import logging
import sys
logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
logging.error( 'example' )


This is starting to look too much like java.  Yuck!

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Keith Thompson]

MRAB  writes:
[...]
> The purpose of stderr is to display status messages, logging and error
> messages, even user prompts, and not mess up the program's actual 
> output. This is important on a *nix system where you might be piping
> the output of one program into the input of another.

I would expect user prompts to be written to stdout, or perhaps to some
system-specific stream like the current tty, not to stderr.  If a
program has user prompts, it probably doesn't make sense to pipe its
output to the input of another.

-- 
Keith Thompson (The_Other_Keith) keith.s.thompso...@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Eryk Sun]

On 1/3/23, c.bu...@posteo.jp  wrote:
>
> If the user request the usage info via "-h" it goes to stdout.

The standard file for application output is sys.stdout. Note that
sys.stdout may be buffered, particularly if it isn't a tty. When a
file is buffered, writes are aggregated and only written to the OS
file when the buffer fills up or is manually flushed.

> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.

The standard file for error messages and other diagnostic information
is sys.stderr. This file should never be buffered.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

Really doesn’t have much to do with Python and very much with standard streams, 
which go back as far as the 1950s. 
https://en.wikipedia.org/wiki/Standard_streams

When a user types -h / --help to a Python argparse script, the output of the 
script is the help message. The standard behavior is to print that and exit, 
regardless of any other arguments that are passed. So its behavior is 
consistent with the concept of standard out.

From: Python-list  on 
behalf of MRAB 
Date: Tuesday, January 3, 2023 at 11:41 AM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

On 2023-01-03 15:35, c.bu...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.

The purpose of stderr is to display status messages, logging and error
messages, even user prompts, and not mess up the program's actual
output. This is important on a *nix system where you might be piping the
output of one program into the input of another.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Richard Damon]



> On Jan 3, 2023, at 10:38 AM, c.bu...@posteo.jp wrote:
> Hello,
> 
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
> 
> The logging module write everything to stderr no matter which logging
> level is used.
> 
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
> 
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
> 
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
> 
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

My thought are, that a “Log” should be a COMPLETE output of what is logged. 
Logged messages are NOT intended to be a message to the current user, but for a 
post operation analysis. Remember, a message sent by logging.info may never be 
seen, as that level of logging may be turned off. (You are providing ways to 
control how much is being set to the log, especially if going to the “screen”).

As such, stderr is a often better spot to send the log than stdout, as the 
program may well be using stdout to interact with the user, and you don’t want 
the interaction cluttered with the logging.

Also, I am fairly sure you can configure the logger to do what you want by just 
configuring two different logging handlers, one for stderr that does just 
errors, and one for stdout that shows all messages. If you don’t want the 
errors on stderr and stdout, just log the errors to a different logger which is 
on an error.* hierarchy and don’t let that hierarchy propagate up to the root 
logger.

 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[MRAB]

On 2023-01-03 15:35, c.bu...@posteo.jp wrote:

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.

Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


The purpose of stderr is to display status messages, logging and error 
messages, even user prompts, and not mess up the program's actual 
output. This is important on a *nix system where you might be piping the 
output of one program into the input of another.

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.


I do not have any special insight into the developers' minds, but here 
is my own reaction.  Logging output is normally not the primary output 
of a program.  For example, you might want to capture the program's 
output and do some processing on it.  In that case you don't want to 
have to parse out the logging text so as to avoid processing it. If not 
to stdout, then where should logging output go?  The only two 
possibilities seem to be stderr, or a known file.


Also, I think it would be confusing to sometimes have logging output go 
to stdout and other times go to stderr.


In the case of argparse, the messages are the intended program output, 
so to me it makes sense that error announcements should go to stderr and 
requests for help information should go to stdout.


There is probably no single right way to do it, but the way it's being 
done makes sense to me, FWIW.




Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.


--
https://mail.python.org/mailman/listinfo/python-list

What should go to stdout/stderr and why Python logging write everything to stderr?

[c.buhtz]

Hello,

this posting isn't about asking for a technical solution. My intention
is to understand the design decision Python's core developers made in
context of that topic.

The logging module write everything to stderr no matter which logging
level is used.

The argparse module does it more differentiated. If arguments are
mandatory but none are given then argparse produce a short version of
the usage info; on stderr. If the user request the usage info via "-h"
it goes to stdout. This is the behavior I would expect.

Why does logging behave different? DEBUG and INFO imho should go to
stdout not stderr.

Of course I could modify the handlers etc. to workaround this. But I
assume that there was something in mind of the Python developers when
they decided that.

My goal is not to divide between the use of print() or logging.info()
in my own code. This would mess up a lot.
-- 
https://mail.python.org/mailman/listinfo/python-list