Re: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments

[aapost]

On 1/4/23 09:42, Dieter Maurer wrote:

aapost wrote at 2023-1-3 22:57 -0500:

...
Consider the following:

from lxml import objectify, etree
schema = etree.XMLSchema(file="path_to_my_xsd_schema_file")
parser = objectify.makeparser(schema=schema, encoding="UTF-8")
xml_obj = objectify.parse("path_to_my_xml_file", parser=parser)
xml_root = xml_obj.getroot()

let's say I have a Version element, that is defined simply as a string
in a 3rd party provided xsd schema




Does your schema include the third party schema?

You might have a look at `PyXB`, too.
It tries hard to enforce schema restrictions in Python code.



Yes, to clarify, they provide the schema, which is what we use, 
downloaded locally. Basically just trying to remain compliant with their 
structures that they already define without reinventing the wheel for 
numerous calls and custom types, and in a way that feels more live 
rather than just checking validity at the end of the edits as if I were 
modifying the XML manually.


Thank you for the suggestion, PyXB works much more like how I envisioned 
working with xml in my head:


>>> xml_root.Version = 1231.32000
pyxb.exceptions_.SimpleTypeValueError: Type 
{http://www.w3.org/2001/XMLSchema}string cannot be created from: 1231.32

>>> xml_root.Version = "1231.32000"

I will have to do some more testing to see how smooth the transition 
back to a formatted document goes, since it creates a variable for all 
possible fields defined in the type, even if they are optional and not 
there in the situational template.


Thanks
--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Chris Angelico]

On Thu, 5 Jan 2023 at 05:06, Barry Scott  wrote:
>
>
> On 03/01/2023 21:24, c.bu...@posteo.jp wrote:
> > Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> >> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
> >
> > But I don't want to make all log levels go to stdout. Just DEBUG and
> > INFO. But this would be a workaround.
> >
> > The main question here is why does Python deciecded to make all logs
> > go to stderr?
> > Maybe I totally misunderstood the intention of logging.info()?! Isn't
> > this the "usual applicaton output"?
> >
> > If not, what messages should go into logging.info()? Can you name me
> > some examples?
>
> Example:
>
> write an app that prints the contents of a file.
>
> The application output is the contents of the file.
>
> The logging might be this when all is working:
>
> INFO About to open 
> INFO Wrote  bytes from 
>
> The logging might be this when there is a problem:
>
> INFO About to open 
> ERROR Failed to open  - 
>
> Does that help?

Notably, the info lines can provide context for an error. For example:

INFO: Loading module XYZ
WARN: Unrecognized module option frobnicate=1
INFO: Using default spamination of 4
ERROR: Insufficient ham to spaminate

This tells a hypothetical (but only slightly hypothetical) story of a
module that was given an incorrect or misspelled option, and which
then failed to load as a consequence. Without the INFO lines, it would
be harder to figure out what the problem was, but they are still part
of logging, not output, and belong in the same place as the warnings
and errors.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


Example:

write an app that prints the contents of a file.

The application output is the contents of the file.

The logging might be this when all is working:

INFO About to open 
INFO Wrote  bytes from 

The logging might be this when there is a problem:

INFO About to open 
ERROR Failed to open  - 

Does that help?

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 03/01/2023 21:24, c.bu...@posteo.jp wrote:

Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:

logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))


But I don't want to make all log levels go to stdout. Just DEBUG and 
INFO. But this would be a workaround.


The main question here is why does Python deciecded to make all logs 
go to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't 
this the "usual applicaton output"?


If not, what messages should go into logging.info()? Can you name me 
some examples?


It is up to you, the designer of an application, to decide how it works.
You will take into account conventions that your users will expect you 
to follow.


If the logging module helps you then use it, but you are not forced by 
logging to
design your app is a particular way. The default behavior of the logging 
module is
a generally useful default, but its only a default. You are free to 
setup logging to

meet your needs.

Barry


--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Thomas Passin]

On 1/3/2023 10:35 AM, c.bu...@posteo.jp wrote:

The logging module write everything to stderr no matter which logging
level is used.


The OP wrote this, but it's not so by default.  There is a HOW-TO page 
that explains this and how to get the logging package to log everything 
to a file, along with many other useful bits of information -


https://docs.python.org/3/howto/logging.html

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-03 21:24:20 +, c.bu...@posteo.jp wrote:
> The main question here is why does Python deciecded to make all logs go to
> stderr?

It doesn't.

The Python logging system provides a plethora of handlers to log to lots
of different places. Only the StreamHandler defaults to writing to
stderr, and even that can log to any stream - just pass it as an
argument.

If none of existing handlers does what you want you can always write
your own.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Peter J. Holzer]

On 2023-01-04 16:46:37 +1100, Chris Angelico wrote:
> On Wed, 4 Jan 2023 at 15:11, Eryk Sun  wrote:
> > On 1/3/23, Chris Angelico  wrote:
> > > writing the FD is the same as using stdout
> >
> > Except stdout may be buffered.

Maybe I'm overly lax in my terminology, but I frequently use the term
"stdout" as a shorthand for "standard output" which refers to file
descriptor 1, not just to the C FILE* of that name. Or Python's
sys.stdout which is a different thing again (but also is a frontend to
FD 1).

> > One should probably flush the buffer
> > before each raw write to the file descriptor.
> 
> FDs can also be buffered.

That depends on what you mean by "buffered". A write to an fd referring
to a file on a disk will not usually affect the disk, true. But it will
be immediately visible to other processes reading that file.

A write to a terminal attached via a serial cable will also be written
to a buffer first (in the OS or maybe inside the UART), but it will
be sent to the terminal as fast as the hardware allows.

So. sure, the writes are "buffered", but is that meaningful in this
context?

Reading from a terminal also often involves a buffer, but in a very
different sense: Here the OS buffers each line to implement line editing
functions. If the kernel didn't do that each application which could
possibly read from a terminal would have to implement that itself. and
it was decided back in the 1970's that this wasn't a good use of
resources. (Of course these days many programs which didn't do that
originally (like shells) do implement line editing (often via
readline).)

> If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.

I don't think fsync() will have an effect on an FD connected to a
terminal (or a pipe or a socket).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 06:46, Chris Angelico wrote:

I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1"


C++ has this feature:

Quote from https://en.cppreference.com/w/cpp/io/cin

"Once |std::cin| is constructed, std::cin.tie() returns &std::cout 
, and likewise, 
std::wcin.tie() returns &std::wcout 
. This means that any 
formatted input operation on |std::cin| forces a call to std::cout 
.flush() if any characters are 
pending for output."


Barry

--
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Barry Scott]

On 04/01/2023 02:26, Chris Angelico wrote:

Reading/writing the FD is the same as using stdout (technically you'd
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.


I think the logic is more like checking that stdin is a tty then using
the tty it to read and write the tty. This works with stdout redirected.
Also stdin is likely only open with read access.

For example the way to prevent ssh-add from prompting in the terminal is 
this:


$ ssh-add ~/.ssh/id_rsa https://mail.python.org/mailman/listinfo/python-list

Re: No solution for "--verbose" (on stdout) output in Pythonds standard library?

[Richard Damon]

> On Jan 4, 2023, at 8:56 AM, c.bu...@posteo.jp wrote:
> 
> Hello,
> 
> first I have to say that in my current and fresh humble opinion the
> often seen "--verbose" switch in command line applications should
> affect only the messages given to the users. This means messages on
> "stdout". That is what this question is about.
> 
> The logging module is not an option because it works on stderr and is
> not intended to offer messages for the user but just for the system and
> its admins via logs (log-files, syslog, stderr redirection, ...).
> 
> Using logging handlers redirecting to stdout are considered as
> workarounds by me and also not an option.
> 
> This is not only my opinion but also part of the Python documentation:
> https://docs.python.org/3/howto/logging.html#when-to-use-logging
> 
> I'm so detailed here about that difference between stdout and stderr
> because in the wild (e.g. on StackOverflow) you often find "use logging
> log levels" as a solution for that problem, which IMHO isn't one.
> 
> Now the question:
> From my research on the docs it seems there is no feature in the
> standard library which could help me to implement "--verbose" or
> multiple verbosity levels like "-vvv"?
> I found some workarounds/hacks.
> https://stackoverflow.com/q/5980042/4865723
> But my experience with Python as a Swiss knife is that there is always
> a standard solution for such basic and often reinvented things. I won't
> believe that each Python developer writes its own verbose feature. ;)
> -- 
> https://mail.python.org/mailman/listinfo/python-list

First, I would say you are incorrect that Payton ALWAYS has a standard solution 
for “basic” problems, as some problems (like this) aren’t actual that easy to 
just provide a solution.

“Verbosity” levels, need to be defined by an application, so can’t just be 
provided by Python, but WILL need some effort on by the programmer.

Second, Stack Overflow is NOT a good source of “Truth” about things, it is just 
a source of what people on Stack Overflow think is truth, which is up to the 
reader to decide if it is actually usable.

Now, it turns out that you CAN use argparse (or similar libraries) and logging 
(or similar libraries) to implement a form of verbosity.

Start off by default to NOT have logging enabled (or only enable for “High” 
level messages).

Different levels of Verbosity can enable lower levels of logging, and perhaps 
move the logging from the “default” of stderr to stdout, all of these are 
possible with these libraries.

Whether this is an “abuse” of the logging library or not is up to you are the 
programmer to decide, no one is making you do it that way, or saying that is 
the way it should be done, just a way it COULD be done.

Personally, I might consider the logging module for this, or I might just add 
print statements conditioned on a verbosity level set by the argument parsing 
module I was using.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: No solution for "--verbose" (on stdout) output in Pythonds standard library?

[Eryk Sun]

On 1/4/23, c.bu...@posteo.jp  wrote:
>
> often seen "--verbose" switch in command line applications should
> affect only the messages given to the users. This means messages on
> "stdout". That is what this question is about.

Is this additional context information such as help and definitions?
If it's anything to do with diagnostics and telemetry, then logging to
stderr is normal. In my experience the verbosity level typically
applies to the latter. I might step it up as I delve deeper into a
problem and need to examine specific details.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python - working with xml/lxml/objectify/schemas, datatypes, and assignments

[Dieter Maurer]

aapost wrote at 2023-1-3 22:57 -0500:
> ...
>Consider the following:
>
>from lxml import objectify, etree
>schema = etree.XMLSchema(file="path_to_my_xsd_schema_file")
>parser = objectify.makeparser(schema=schema, encoding="UTF-8")
>xml_obj = objectify.parse("path_to_my_xml_file", parser=parser)
>xml_root = xml_obj.getroot()
>
>let's say I have a Version element, that is defined simply as a string
>in a 3rd party provided xsd schema
>
>

Does your schema include the third party schema?

You might have a look at `PyXB`, too.
It tries hard to enforce schema restrictions in Python code.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: No solution for "--verbose" (on stdout) output in Pythonds standard library?

[Weatherby,Gerard]

A couple options. The -vvv is more of Linux thing rather than Pythonic, to my 
way of thinking.



import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-v',action='store_true')
parser.add_argument('-vv',action='store_true')
parser.add_argument('-vvv',action='store_true')
args = parser.parse_args()

parser = argparse.ArgumentParser()
parser.add_argument('-v','--verbose',type=int,default=0,nargs='?',help="verbose 
level")
args = parser.parse_args()
level = args.verbose if args.verbose is not None else 0
print(level)


Personally I just do:

import argparse
import logging


logging.basicConfig()
parser = argparse.ArgumentParser()
parser.add_argument('-l', '--loglevel', default='WARN', help="Python logging 
level")
args = parser.parse_args()
logger = logging.getLogger(__name__)
logger.setLevel(getattr(logging,args.loglevel))

From: Python-list  on 
behalf of c.bu...@posteo.jp 
Date: Wednesday, January 4, 2023 at 8:55 AM
To: python-list@python.org 
Subject: No solution for "--verbose" (on stdout) output in Pythonds standard 
library?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

Hello,

first I have to say that in my current and fresh humble opinion the
often seen "--verbose" switch in command line applications should
affect only the messages given to the users. This means messages on
"stdout". That is what this question is about.

The logging module is not an option because it works on stderr and is
not intended to offer messages for the user but just for the system and
its admins via logs (log-files, syslog, stderr redirection, ...).

Using logging handlers redirecting to stdout are considered as
workarounds by me and also not an option.

This is not only my opinion but also part of the Python documentation:
https://urldefense.com/v3/__https://docs.python.org/3/howto/logging.html*when-to-use-logging__;Iw!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLfNcgTsq$

I'm so detailed here about that difference between stdout and stderr
because in the wild (e.g. on StackOverflow) you often find "use logging
log levels" as a solution for that problem, which IMHO isn't one.

Now the question:
>From my research on the docs it seems there is no feature in the
standard library which could help me to implement "--verbose" or
multiple verbosity levels like "-vvv"?
I found some workarounds/hacks.
https://urldefense.com/v3/__https://stackoverflow.com/q/5980042/4865723__;!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLU6LbUZG$
But my experience with Python as a Swiss knife is that there is always
a standard solution for such basic and often reinvented things. I won't
believe that each Python developer writes its own verbose feature. ;)
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!gctSoo2C-FjIeI4wfVetRVylYZ-X1te71-Q05ylEpJ_2XICGGoFbXFjrm02smi-UKx0H2EbiEXiJLYWBuS4g$
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: No solution for "--verbose" (on stdout) output in Pythonds standard library?

[Chris Angelico]

On Thu, 5 Jan 2023 at 00:54,  wrote:
>
> Hello,
>
> first I have to say that in my current and fresh humble opinion the
> often seen "--verbose" switch in command line applications should
> affect only the messages given to the users. This means messages on
> "stdout". That is what this question is about.
>
> The logging module is not an option because it works on stderr and is
> not intended to offer messages for the user but just for the system and
> its admins via logs (log-files, syslog, stderr redirection, ...).

We've already explained in the other thread that these kinds of logs -
which are EXACTLY what a --verbose switch usually covers - belong on
stderr. I don't understand the problem here.

But have you looked into argparse? I mean, you did say you researched
the docs, so surely you must have found it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

No solution for "--verbose" (on stdout) output in Pythonds standard library?

[c.buhtz]

Hello,

first I have to say that in my current and fresh humble opinion the
often seen "--verbose" switch in command line applications should
affect only the messages given to the users. This means messages on
"stdout". That is what this question is about.

The logging module is not an option because it works on stderr and is
not intended to offer messages for the user but just for the system and
its admins via logs (log-files, syslog, stderr redirection, ...).

Using logging handlers redirecting to stdout are considered as
workarounds by me and also not an option.

This is not only my opinion but also part of the Python documentation:
https://docs.python.org/3/howto/logging.html#when-to-use-logging

I'm so detailed here about that difference between stdout and stderr
because in the wild (e.g. on StackOverflow) you often find "use logging
log levels" as a solution for that problem, which IMHO isn't one.

Now the question:
>From my research on the docs it seems there is no feature in the
standard library which could help me to implement "--verbose" or
multiple verbosity levels like "-vvv"?
I found some workarounds/hacks.
https://stackoverflow.com/q/5980042/4865723
But my experience with Python as a Swiss knife is that there is always
a standard solution for such basic and often reinvented things. I won't
believe that each Python developer writes its own verbose feature. ;)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[2QdxY4RzWzUUiLuE]

On 2023-01-04 at 12:02:55 +,
"Weatherby,Gerard"  wrote:

> Dealing with stdout / stderr is bash is just syntax. I don’t always
> remember it off the top of my head but … stackoverflow.com.

https://tldp.org/LDP/abs/html/io-redirection.html

> On Linux at least it’s possible to pipe to arbitrary streams, it
> seems. The following uses bash to write “Hi” to the file “third”
> opened by Python. I determined the file was 5 empirically.

Empirically?

> import os
> import subprocess
> command= 'echo Hi >/dev/fd/5'
> fd = os.open("third",os.O_WRONLY|os.O_CREAT)

command = f"echo Hi >/dev/fd/{fd}"

Or:

command = f"echo Hi >&{fd}"

> os.set_inheritable(fd,True)
> 
> subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)

By the time I'm that far down that path, I usually write a separate
function to call fork, open, close, and exec myself (old habits die
hard).  In the end, there's no reward for writing and maintaining shell
programs in Python.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What should go to stdout/stderr and why Python logging write everything to stderr?

[Weatherby,Gerard]

Dealing with stdout / stderr is bash is just syntax. I don’t always remember it 
off the top of my head but … stackoverflow.com.

On Linux at least it’s possible to pipe to arbitrary streams, it seems. The 
following uses bash to write “Hi” to the file “third” opened by Python. I 
determined the file was 5 empirically.


import os
import subprocess
command= 'echo Hi >/dev/fd/5'
fd = os.open("third",os.O_WRONLY|os.O_CREAT)
os.set_inheritable(fd,True)

subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)

From: Python-list  on 
behalf of Michael Torrie 
Date: Tuesday, January 3, 2023 at 5:18 PM
To: python-list@python.org 
Subject: Re: What should go to stdout/stderr and why Python logging write 
everything to stderr?
*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***


Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams.  Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.




--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$
-- 
https://mail.python.org/mailman/listinfo/python-list