Re: Using astype(int) for strings with thousand separator

[Thomas Jollans]

On 14.11.21 16:41, Mahmood Naderan via Python-list wrote:

Hi

While reading a csv file, some cells have values like '1,024' which I mean they 
contains thousand separator ','. Therefore, when I want to process them with

   row = df.iloc[0].astype(int)



If you are reading a CSV with pandas.read_csv, that function takes a 
parameter 'thousands' that should do what you're looking for (see 
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)




I get the following error

   ValueError: invalid literal for int() with base 10: '1,024'


How can I fix that? Any idea?




--
https://mail.python.org/mailman/listinfo/python-list

Re: Confusing error message: lambda walruses

[Thomas Jollans]

On 06/10/2021 23:53, Chris Angelico wrote:

On Thu, Oct 7, 2021 at 8:51 AM Thomas Jollans  wrote:

On 03/10/2021 01:39, Chris Angelico wrote:

Using assignment expressions in lambda functions sometimes works, but
sometimes doesn't.

Does this commit by a certain Chris Angelico help clear things up?

https://github.com/python/peps/commit/f906b988b20c9a8e7e13a2262f5381bd2b1399e2

No, because the examples I gave don't fit into that :) I was aware of
what the PEP originally said.

If you try to align the examples with the descriptions in the PEP,
you'll find that they don't all match.

ChrisA



The issue closed by that commit is interesting, if nothing else:

Guido van Rossum wrote:

The PEP says very little about lambda. I feel the following two 
examples should both be valid:


foo(f := lambda: 42, flag=True)
foo(lambda: f := 42, flag=True)

Chris Angelico wrote:

The second example is kinda bizarre though; it's going to create a 
fairly useless name binding within the lambda function. (Unless you 
want to give lambda functions the same magic as comprehensions, making 
the name f exist in the same scope where foo is being run.) So I would 
be okay with the first example doing the obvious thing, and the second 
one requiring parentheses lambda: (f := 42) for syntactic validity.


I think that at least clears up this part:


# But a SyntaxError if parenthesized like this??

def f(n):

... return (lambda: n := 1)
   File "", line 2
 return (lambda: n := 1)
 ^
SyntaxError: cannot use assignment expressions with lambda

# Oh, and it doesn't actually assign anything.

def f(n):

... return (lambda: (n := 1)), (lambda: n)
...



--
https://mail.python.org/mailman/listinfo/python-list

Re: Confusing error message: lambda walruses

[Thomas Jollans]

On 03/10/2021 01:39, Chris Angelico wrote:

Using assignment expressions in lambda functions sometimes works, but
sometimes doesn't.


Does this commit by a certain Chris Angelico help clear things up?

https://github.com/python/peps/commit/f906b988b20c9a8e7e13a2262f5381bd2b1399e2
--
https://mail.python.org/mailman/listinfo/python-list

Re: matplotlib graph white space

[Thomas Jollans]

On 04/10/2021 10:39, Steve wrote:

I am using the first bar graph listed at this site:
https://matplotlib.org/stable/gallery/index.html

The problem I have is that there is too much white space around the graph.
My data would be better displayed if I could widen the graph into the space
to the right and left of the chart.

Steve

To tweak the amount of padding around the matplotlib axes, you can use 
pyplot.subplots_adjust [1] or Figure.subplots_adjust [2].


In most cases, the tight_layout function/method [3,4] will give you 
sensible settings for the various spacing parameters.


You probably also have to adjust the figure size (see David Lowry-Duda's 
reply) to get whatever effect it is that you want.


[1] 
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.subplots_adjust.html
[2] 
https://matplotlib.org/stable/api/figure_api.html#matplotlib.figure.Figure.subplots_adjust
[3] 
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.tight_layout.html
[4] 
https://matplotlib.org/stable/api/figure_api.html#matplotlib.figure.Figure.tight_layout


--
https://mail.python.org/mailman/listinfo/python-list

Re: The split() function of Python's built-in module has changed in a puzzling way - is this a bug?

[Thomas Jollans]

On 23/04/2021 01:53, Andy AO wrote:

Upgrading from Python 3.6.8 to Python 3.9.0 and executing unit tests
revealed a significant change in the behavior of re.split().

but looking at the relevant documentation — Changelog  and re - Regular expression
operations - Python 3.9.4 documentation

yet no change is found.

number = '123'def test_Asterisk_quantifier_with_capture_group(self):
 resultList = re.split(r'(\d*)', self.number)
 if platform.python_version() == '3.6.8':
 self.assertEqual(resultList,['', '123', ''])

 else:
 self.assertEqual(resultList,['', '123', '', '', ''])



Hi Andy,

That's interesting. The old result is less surprising, but of course 
both are technically correct as the 4th element in the result matches 
your regexp.


The oldest version of Python I had lying around to test is 3.7; that has 
the same behaviour as 3.9.


I suspect that this behaviour is related to the following note in the 
docs for re.split:



Changed in version 3.7: Added support of splitting on a pattern that 
could match an empty string.



(your pattern can match an empty string, so I suppose it wasn't 
technically supported in 3.6?)



-- Thomas





I feel that this is clearly not in line with the description of the
function in the split documentation, and it is also strange that after
replacing * with +, the behavior is still the same as in 3.6.8.

1. why is this change not in the documentation? Is it because I didn’t
find it?
2. Why did the behavior change this way? Was a bug introduced, or was it
a bug fix?


--
https://mail.python.org/mailman/listinfo/python-list

Re: pandas/jupyther notebook?

[Thomas Jollans]

On 07/04/2021 23:32, Jim Byrnes wrote:

linux mint 20
python 3.8
jupyter 1.0.0
jedi 0.18.0

I am teaching myself pandas/jupyter notebooks. The problem I am having 
is tab autocomplete seems to be working erratically.


Googling shows that most people solve autocomplete problems by putting

 import pandas as pd
%config Completer.use_jedi = False


One solution is to downgrade to jedi 0.17.2




in the first cell. I have done that and still have the following problem.

if I type: df = pd.read it will drop down a list box that has 
read_csv in it. If I then type: emp it will fill in 
employees.csv. If I type: df.h type head, but if I type:


df['Gender'] = df['Gender'].ast  it will not complete astype.

I wonder if someone can tell me why this is happening and maybe how to 
fix it.


Thanks,  Jim



--
Dr. Thomas Jollans

m ☎ +49 6201 8759879
e ✉ t...@tjol.eu

--
https://mail.python.org/mailman/listinfo/python-list

Re: Matplotlib scale

[Thomas Jollans]

On 04/04/2021 20:57, Julien Hofmann wrote:

Hi everyone,


I've created a code to run a 2D mapping using matplotlib from a .csv file.
I've tried to set the maximum color (red) of the scale as 80% of the maximum 
value and not as the maximum value of my .csv file.
Does someone know how to modify that?


Most (or all?) matplotlib functions and methods that take a cmap 
argument also take vmin and vmax arguments to specify the maximum and 
minimum values to assign colours to.


https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.contourf.html

So adding vmax=0.8*np.max(z) to your contourf() call should do the trick.

A couple of quick data visualisation tips:

1. if your colour map doesn't cover the full range of data, your colour 
bar should indicate this. Call fig.colorbar(, extend='max').


2. 'jet' is a *terrible* colour map and you should *never* use it. It 
distorts your data, making people see patterns that aren't there, and is 
all but useless in black & white printouts or to the colour-blind.


This seminal talk from 2015 explains why the default Matplotlib colour 
maps are what they are: https://www.youtube.com/watch?v=xAoljeRJ3lU



-- Thomas



I've tried different solution but it doesn't work.

Thanks

import os
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from matplotlib import colorbar, colors
import matplotlib.tri as tri
  
#os.chdir("C:/Users/Julien Hofmann/Desktop/Nano-indentation")

data = pd.read_csv("Cartographie.csv",sep=';')
  
nb_lignes=21

nb_colonnes=27
  
  
fig = plt.figure(figsize=(15,12))

ax = plt.subplot(1,1,1)
x=np.linspace(0,(data["x"][len(data["x"])-1]-data["x"][0])*1000,nb_colonnes)
y=np.linspace(0,(data["y"][len(data["y"])-1]-data["y"][0])*1000,nb_lignes)
X,Y=np.meshgrid(x,y)
  
  
z=np.array(data["Durete"])

triang = tri.Triangulation(data["x"], data["y"])
interpolator = tri.LinearTriInterpolator(triang, z)
Xi, Yi = np.meshgrid(x, y)
zi = interpolator(Xi, Yi)
cntr1 = ax.contourf(x, y, z.reshape(nb_lignes,nb_colonnes), levels=150, 
cmap="jet")
cbar = fig.colorbar(cntr1, ax=ax)
ax.axis('on')



--
Dr. Thomas Jollans

☎ +49 6201 8759879
✉ t...@tjol.eu

--
https://mail.python.org/mailman/listinfo/python-list

Re: python documentation

[Thomas Jollans]

On 27/03/2021 06:20, pyt...@blackward.eu wrote:

Chris,

you seem to imply, that I have compiled said versions without reason 
and that the same would be possible on basis of Python 3 - which is 
simply not true. Maybe you are not enough acquainted with Qt and 
belonging libraries alike PyQtGraph. Maybe you are just not willing to 
see / accept these arguments.


By the way, some months ago I started trying to migrate to Python 3 
and gave up in favor of creating said compilation. Compatibility of 
Python and its Packages decreased with V3 significantly. A whole lot 
of minor and major incompatibilities between your subversions and 
belonging packages. This was one reason, why Java took the route to 
its own death.


*rolls eyes*

I know PyQtGraph reasonably well and this is the first time I've ever 
heard of anybody using it on Python 2. I mean, I imagine it once worked 
on Python 2 and probably still does, but all of these package have had 
perfectly good Python 3 support for many, many years.



- Thomas


--
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Thomas Jollans]

On 19/03/2021 20:33, dn via Python-list wrote:

On 20/03/2021 07.49, Grant Edwards wrote:

On 2021-03-19, MRAB  wrote:

You want English "man's" to become "Man's", but French "l'homme" to
become "L'Homme". It's language-dependant.

In English, certain words are not capitalized in titles unless they're
the first word in the title (short articles and prepositions), and
.title() doesn't get that right either:


"the man in the grey flannel suit".title()

'The Man In The Grey Flannel Suit'

should be

'The Man in the Grey Flannel Suit'


To be fair, aren't book-titles* a (formalised) sub-set of the English
language?


From a quick scan of my (medium-sized) bookshelf, most publishers seem 
to agree that the thing to do is set the title in all caps. Of the few 
(English-language) books I have with any lower-case letters on the spine 
at all, most seem to follow the same general sort of rules for what 
words to capitalize, but ‘Last Chance To See’, capital T, by Douglas 
Adams, is one exception.


Of the others, I noticed that ‘The Life and Times Times of The 
Thunderbolt Kid’ by Bill Bryson has an interesting choice of 
capitalization which is perfectly logical but certainly not the only option.




--
https://mail.python.org/mailman/listinfo/python-list

Re: yield from () Was: Re: weirdness with list()

[Thomas Jollans]

On 03/03/2021 01:01, Cameron Simpson wrote:

On 02Mar2021 15:06, Larry Martell  wrote:

I discovered something new (to me) yesterday. Was writing a unit test
for generator function and I found that none of the function got
executed at all until I iterated on the return value.

Aye. Generators are lazy - they don't run at all until you ask for a
value.

By contrast, this is unlike Go's goroutines, which are busy - they
commence operation as soon as invoked and run until the first yield
(channel put, I forget how it is spelled now). This can cause excessive
CPU utilisation, but it handle for _fast_ production of results. Which
is a primary goal in Go's design.

Cheers,
Cameron Simpson 




I've been learning a bit more JavaScript recently (I know, I know, 
that's no fun) and I think that's the main practical difference between 
JavaScript's async functions, which are scheduled even if nobody awaits 
on them, and Python async functions which are just funky generators and 
therefore scheduled only when somebody awaits their result.




--
https://mail.python.org/mailman/listinfo/python-list

Re: Application problems

[Thomas Jollans]

On 11/03/2021 15:06, Anssi Saari wrote:

Thomas Jollans  writes:


On 10/03/2021 21:50, Mats Wichmann wrote:

For the first one, don't feel too bad, this ("opening the normal
python") seems to be biting a lot of people recently


I wonder why. Python's installation process isn't any different from
most other Windows software released the past 25-ish years. Is it
possible that Windows 10's search feature sometimes makes poor
choices, and typing "python" just brings up the wrong thing?

I'm thinking maybe it's people who've never installed anything? Since
for Facebook or whatever you only need a browser...


Plus now there's the Microsoft Store for a lot of things, yeah

--
https://mail.python.org/mailman/listinfo/python-list

Re: Application problems

[Thomas Jollans]

On 10/03/2021 21:50, Mats Wichmann wrote:


For the first one, don't feel too bad, this ("opening the normal 
python") seems to be biting a lot of people recently



I wonder why. Python's installation process isn't any different from 
most other Windows software released the past 25-ish years. Is it 
possible that Windows 10's search feature sometimes makes poor choices, 
and typing "python" just brings up the wrong thing?


(I just tested it on a clean VM and that's not what happens, but maybe 
for some people? I dunno)


--
https://mail.python.org/mailman/listinfo/python-list

Re: How to create both a c extension and a pure python package

[Thomas Jollans]

On 10/03/2021 20:57, Mats Wichmann wrote:

On 3/10/21 11:56 AM, Thomas Jollans wrote:

On 10/03/2021 18:42, Marco Sulla wrote:

On Wed, 10 Mar 2021 at 16:45, Thomas Jollans  wrote:

Why are you doing this?

If all you want is for it to be possible to install the package from
source on a system that can't use the C part, you could just declare
your extension modules optional

Because I want to provide (at least) two wheels: a wheel for linux
users with the C extension compiled and a generic wheel in pure python
as a fallback for any other architecture.


What's wrong with sdist as a fallback rather than a wheel?

That has the added benefit of people on other architectures have the 
opportunity to use the extension module if they have a compiler and 
the necessary libraries and headers installed...


Doesn't that mean nasty failures for those don't have the correct 
build setup (like almost every Windows user on the planet)?   This 
isn't a snide question, I'm actually interested in solving roughly the 
same problem as the OP.


I believe that this is pretty much exactly the problem that the 
"optional" flag for extensions solves.


docs: "specifies that a build failure in the extension should not abort 
the build process, but simply skip the extension."


I would assume this refers to *any* build failure (including a missing 
compiler), so people without a proper build system might get error 
messages, but should still get a working package (assuming of course the 
C extensions is *actually* optional, like an accelerator module).


But I don't have experience with this myself so take what I say with a 
grain of salt.



- Thomas



--
https://mail.python.org/mailman/listinfo/python-list

Re: How to create both a c extension and a pure python package

[Thomas Jollans]

On 10/03/2021 18:42, Marco Sulla wrote:

On Wed, 10 Mar 2021 at 16:45, Thomas Jollans  wrote:

Why are you doing this?

If all you want is for it to be possible to install the package from
source on a system that can't use the C part, you could just declare
your extension modules optional

Because I want to provide (at least) two wheels: a wheel for linux
users with the C extension compiled and a generic wheel in pure python
as a fallback for any other architecture.


What's wrong with sdist as a fallback rather than a wheel?

That has the added benefit of people on other architectures have the 
opportunity to use the extension module if they have a compiler and the 
necessary libraries and headers installed...





If I make the extension optional, as far as I know, only one wheel is
produced: the wheel with the extension if all is successful, or the
pure py wheel.



--
https://mail.python.org/mailman/listinfo/python-list

Re: How to create both a c extension and a pure python package

[Thomas Jollans]

On 09/03/2021 23:42, Marco Sulla wrote:

As title. Currently I ended up using this trick in my setup.py:


if len(argv) > 1 and argv[1] == "c":
 sys.argv = [sys.argv[0]] + sys.argv[2:]
 setuptools.setup(ext_modules = ext_modules, **common_setup_args)
else:
 setuptools.setup(**common_setup_args)


So if I pass "c" as the first argument of ./setup.py , the c extension
is builded, otherwise the py version is packaged.

Is there not a better way to do this?



Why are you doing this?

If all you want is for it to be possible to install the package from 
source on a system that can't use the C part, you could just declare 
your extension modules optional, with the "optional" argument to 
setuptools.Extension. see 
(https://docs.python.org/3/distutils/apiref.html#distutils.core.Extension)



- Thomas


--
https://mail.python.org/mailman/listinfo/python-list

Re: Best practices regarding PYTHONPATH

[Thomas Jollans]

On 09/03/2021 22:52, Cameron Simpson wrote:

On 09Mar2021 05:00, Larry Martell  wrote:

Which is considered better? Having a long import path or setting PYTHONPATH?

For example, in a project where 50% of the imports come from the same top
level directory is it better to add that dir to the path or reference it in
the import statements?

All the below is personal opinion.

I'd be leaving the $PYTHONPATH alone - I tweak it to access the required
libraries, but not to change their dotted module paths.

For example, I include ~/lib/python in my personal environment to access
my personal modules, but I don't include
~/lib/python/cs/app/someapp/subpackage in order to shorten
"cs.app.someapp.subpackage.foo" to just "foo".

This is largely to avoid accidental shadowing of other modules. For
example, supposing "foo" above were the subpath "os.path". Yes,
contrived, but that's the flavour of problem I'm avoiding.

I think I'd be ok with it provided I didn't go too far down. Eg, if all
my "someapp"s were distinctively named I could be persuaded to use
~/lib/python/cs/app in the $PYTHONPATH, allowing
"someapp.subpackage.foo". But I'd still be reluctant.


If you have a bunch of different packages you want to think of as "top 
level" but that live in different places for organizational reasons 
(e.g. someapp in ~/lib/python/cs/app/someapp and thatlib in 
~/lib/python/cs/lib/util/thatlib), I'd advocate treating them as proper 
Python packages with their own setup.py and everything, and installing 
them properly.


Editable installs (setup.py develop or pip install -e .) are great for this.

Personally when this is impractical for some reason I prefer creating 
*.pth files in site-packages to messing with PYTHONPATH as I find this 
easier to maintain, especially if you ever end up using different Python 
versions or different virtual environments.


Just my 0.02 €.




If the project modules are tightly bound relative imports can get you a
fair way. Certainly within a package I do a lot of:

 from .fixtures import these_things

to grab from the adjacent "fixtures.py" file instead of:

 from project.submodule.fixtures import these_things

And I'm usually happy to go up an additional level:

 from ..package2.fixtures import those_things

Somewhere around 3 dots I start to worry about presuming too much, but
that is an arbitrary decision based on the discipline (or lack of it) in
the project naming.

Cheers,
Cameron Simpson 



--
https://mail.python.org/mailman/listinfo/python-list

Re: Choosable dependency

[Thomas Jollans]

On 06/03/2021 12:00, Manfred Lotz wrote:

Let us say I have a package which reads a TOML file.

I want to give the user of my package the choice to decide if he wants
to use the toml, tomlkit or rtoml package.

So, in case the user chose to use rtoml then there should be an import
only for rtoml, aso.

How could I achieve this?

I expect in this case the best thing to do is to simply try to import 
your favourite TOML module (rtoml or what have you), and if that doesn't 
work import a fallback – accepting that there might be two TOML parser 
modules loaded in the same process in some cases.


But this dilemma is fairly common in the Qt GUI world, as there are two 
competing Python bindings for the Qt framework (and multiple 
nearly-compatible versions of both) and in that case all packages need 
to use the same one. So you could have a look at what modules depending 
on Qt do.


For example:

https://pyqtgraph.readthedocs.io/en/latest/how_to_use.html#pyqt-and-pyside

PyQtGraph checks if any supported Qt bindings are already imported and 
uses those. If not, it imports its own favourite.


The rules are fairly complex in their case (they're supporting 6 
almost-but-not-quite-compatible backends), so they have an internal 
wrapper module


https://github.com/pyqtgraph/pyqtgraph/blob/master/pyqtgraph/Qt.py

and do something like

from . import Qt

in every file.

Matplotlib does it the other way around:

https://matplotlib.org/stable/api/matplotlib_configuration_api.html#matplotlib.use

there the user imports matplotlib and then optionally calls a function 
to select a backend. Then every function within matplotlib that uses the 
backend has to defer to some wrapper (not sure where or how)



Hope that helps

Thomas


--
Dr. Thomas Jollans

e ✉ t...@tjol.eu

--
https://mail.python.org/mailman/listinfo/python-list

Re: editor recommendations?

[Thomas Jollans]

On 26/02/2021 15:51, Ethan Furman wrote:
>
> So, what's the state-of-the-art with regards to editors supporting
> dark color themes?

You’re in luck, “Dark Mode” is very much en vogue these days. Most
modern programmer's editors (meaning editors that think of themselves as
modern) are either dark by default or at least come with a fully dark theme.

I mostly use Sublime Text [€€€] myself. Visual Studio Code is pretty
similar, but a fair bit more resource-intensive thanks to being built on
Electron. They're both dark-by-default, fairly configurable and have
communities obsessed with (usually dark) colour schemes. And they both
have loads of plugins which can slow your editor down and give you smart
completion and tooltips with docstrings if you like that sort of thing.
Atom is in the same general category as VS Code.

Spacemacs sounds like it could be fun. ¯\_(ツ)_/¯

- Thomas


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python 2.6: Need some advise for installing modules on a legacy system

[Thomas Jollans]

On 24/02/2021 14:13, Antoon Pardon wrote:
> I need to do some development on this legacy system. It only runs
> python2.6 and there is little hope of installing an other version. How
> can I best proceed to install modules for working with mysql and ldap?
>

The answer very much depends on the operating system. If it's a Linux
system with a working package manager, chances are most common modules
were packaged by the distribution vendor, and your best bet is probably
to get the RPMs or deb packages from the corresponding package archive.
There usually is one.

Otherwise, as Dan suggested, find the source tar.gz (or zip, or
whatever) file for the last version that supported python 2.6 on the
corresponding project's website or on PyPI and install the packages (and
all their dependencies) using the setup.py scripts. If you're on
Windows, you'll probably need Visual Studio 2008 or at least the Visual
C++ compiler version 9.0 (see
<https://wiki.python.org/moin/WindowsCompilers#Microsoft_Visual_C.2B-.2B-_9.0_standalone:_Visual_C.2B-.2B-_Compiler_for_Python_2.7_.28x86.2C_x64.29>)

That being said, I am curious: if you can install Python modules, why
can't you install a newer Python interpreter? Are you running Windows 2000?

Have fun I guess

Thomas


-- 
Dr. Thomas Jollans

✉ t...@tjol.eu

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 2.7 and 3.9

[Thomas Jollans]

On 16/02/2021 22:16, Ethan Furman wrote:
Kevin, please reply to the list (preferably Reply-to-list, or 
Reply-all), that way others can chime in with help.


On 2/16/21 12:55 PM, Kevin M. Wilson wrote:

Windows 7 OS, and typically run in conjunction with testing SSD', as 
for stand alone scripts.
Those require: python BogusFile.py. I too was expecting users to 
type: python my_Script.py!


On Windows, you should use the py.exe launcher:

py -2 python2_script.py

to run an old script, and

py -3 python3_script.py

or

py python3_script.py

to launch a new script. AKAIK, the launcher is always installed with 
recent versions of Python on Windows.



You could set up the PATH such that 'python' is python 2.7, and 'py' 
calls python 3.x. Have a look at the docs to figure about what other 
options py.exe gives you (such as shebang lines)


Docs: 
https://docs.python.org/3/using/windows.html#python-launcher-for-windows


PEP: https://www.python.org/dev/peps/pep-0397


-- Thomas



--
https://mail.python.org/mailman/listinfo/python-list

Re: IDE tools to debug in Python?

[Thomas Jollans]

On 27/01/2021 19:32, flaskee via Python-list wrote:
> 
> While print() is groovy and all,
> if anyone runs across a non-pdb python debugger (standalone or IDE-based)
> please let me know.
> 
> I too was blessed with IDE-based debugging (in the 90's!)
>  * where you can set break point(s);
>  * have the program stop right before a suspected failure point;
>  * check the contents of ALL variables, and choose whether to restart;
>  * or skip a few lines before restarting;
>  * or change a variable (hot, move back a few lines and restart, etc.
>  * Some, would even let you alter the code a bit before restarting.
> 
> I too, miss this.
> 
> 
> Hopefully I did not miss someone mentioning
> such a python tool in the prior thread.

This comes up every now and again. There are a number of visual
debugging tools for Python, but a lot of people don't bother with them.
I don't, though I've been meaning to have a closer look into the options.

https://wiki.python.org/moin/PythonDebuggingTools

There are full-fat IDEs like PyCharm, Eclipse PyDev, Komodo, Wing, or
Visual Studio. There's Spyder, which is reasonably popular in the
scientific Python ecosystem. I think there are VSCode plugins for Python
debugging.

And there is even an experimental debugger for Jupyter now!

Just to give you some pointers as to where to look.

-- Thomas

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Letter replacer - suggestions?

[Thomas Jollans]

On 07/12/2020 16:48, Bischoop wrote:

I worked on my wee script that replaces a letters: https://bpa.st/OYBQ .
I would like to have some suggestions about the code from more
experienced programmers, the code does work and do its job but perhaps
could work in a better way.

Thanks



Sure!

First of all, the code could be simplified by using the replace() method 
of the str class 
(https://docs.python.org/3/library/stdtypes.html#str.replace).


Now a few comments on the code specifically:

    while True:

What is this for? It looks like all it does is cause the program to get 
caught in an infinite loop if there's an exception...


    for element in word_list:
    for x in change_this_list:

You can loop over strings as well - there's no need to convert them to 
lists.


    to = word_list.index(element)

There's a nice trick to get the index while looping: write

    for (idx, character) in enumerate(word):

then you don't have to use the index method. This also works if a 
character appears multiple times in the word, in which case I think your 
code will fail (but I haven't tried it)


(Obviously apply this to the other loop as well, mutadis mutandis, to 
get rid of both index() calls)


    word_list[to] = replace_with_list[replace]

Okay, this is something you can't do with a str because they're 
immutable. Generally I'd avoid modifying the thing you're looping over; 
it works in this case when you're replacing elements of a list, but it 
probably won't do what you want if you're deleting or adding items.


I'd put

new_word_list = []

somewhere at the top, and build it up element by element with 
new_word_list.append(). But what you're doing is fine too as long as you 
keep in mind that changing the thing you're looping over can be 
dangerous in other cases.



Hope this helps


Thomas


--
https://mail.python.org/mailman/listinfo/python-list

Re: how to plot the FFT of a list of values

[Thomas Jollans]

On 05/12/2020 23:08, Christian Gollwitzer wrote:

Am 05.12.20 um 18:16 schrieb Boris Dorestand:

I have 16 values of the period sequence 1, 2, 4, 8, 1, 2, 4, 8, ...  I
compute its fourier transform using


from scipy import fft, ifft
x = [1,2,4,8,1,2,4,8]
fft(x)

array([ 30. +0.j,   0. +0.j,  -6.+12.j,   0. +0.j, -10. +0.j, 0. +0.j,
 -6.-12.j,   0. +0.j])

Now how can I plot these values?  I would like to plot 16 values.  What
do I need to do here?  Can you show an example?



Usually, for the FFT of real input data, you plot only the magnitude 
or square of the complex array, and usually on a logscale. So:


import pylab


Don't use pylab.

https://matplotlib.org/api/index.html#module-pylab

Use matplotlib.pyplot directly instead. "plt" is a popular shorthand:

from matplotlib import pyplot as plt

#...

plt.semilogy(...) # or plt.plot, etc.


- Thomas



import numpy as np

fx = fft(x)

pylab.semilogy(np.abs(fx))
pylab.show()



Christian





--
https://mail.python.org/mailman/listinfo/python-list

Re: Saying bye bye to Python 2

[Thomas Jollans]

On 11/01/2020 00:16, tommy yama wrote:

As many know, python 2 was retired. 🐍
This means imminent migration to 3 will be a must ?


Upgrading to Python 3 has been a "bloody well should" for many, many 
years now.



Though in a shock announcement a few weeks ago the end of Python 2 was 
delayed AGAIN (this time only by a few months, but still)


https://pyfound.blogspot.com/2019/12/python-2-sunset.html



I thought that upgrading is not that simple.


thanks !


--
https://mail.python.org/mailman/listinfo/python-list

Re: How to specific multiple dtypes in numpy.ndarray?

[Thomas Jollans]

On 19/12/2019 11.52, lampahome wrote:
> I meet performance is low when I use struct.unpack to unpack binary data.
>
> So I tried to use numpy.ndarray
> But meet error when I want to unpack multiple dtypes
>
> Can anyone teach me~
>
> Code like below:
> # python3
> import struct
> import numpy as np
> s1 = struct.Struct("@QIQ")
> ss1 = s1.pack(1,11,111)
> np.ndarray((3,), [('Q','I','Q')], ss1)
> # ValueError: mismatch in size of old and new data-descriptor.

A numpy array always has ONE dtype for ALL array elements.

If you read an array of structs, you can define a structured type, where
each element of your struct must have a name.

The error you're seeing is (as you know) because you're not setting up
your dtype in the right way. Let's fix it:

> In [2]: np.dtype([('Q', 'I',
> 'Q')])
>  
>
> ---
> ValueError    Traceback (most recent call
> last)
>  in 
> > 1 np.dtype([('Q', 'I', 'Q')])
>
> ValueError: mismatch in size of old and new data-descriptor
>
> In [3]: np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3',
> 'Q')])   
> Out[3]: dtype([('field1', '
> In [4]:   
>
>

... and now let's put it all together!

s1 = struct.Struct("@QIQ")
ss1 = s1.pack(1,11,111)
struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3', 'Q')])
a = np.frombuffer(ss1, dtype=struct_dtype)

I'm using the frombuffer() function deliberately so I don't have to
figure out the shape of the final array (which is (1,), not (3,), by the
way).

And hey presto: it raises an exception!

> ValueError: buffer size must be a multiple of element size

Your example shows a difference between the default behaviour of numpy's
structured dtype and the struct module: packing! By default, numpy
structured dtypes are closely packed, i.e. nothing is aligned to useful
memory boundaries.

struct_type.itemsize == 20

The struct module, on the other hand, tries to guess where the C
compiler would put its padding.

len(ss1) == 24

We can tell numpy to do the same:

struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3',
'Q')], align=True)

and then

a = np.frombuffer(ss1, dtype=struct_dtype)

works and produces

array([(1, 11, 111)],
  dtype={'names':['field1','field2','field3'],
'formats':['https://mail.python.org/mailman/listinfo/python-list

Re: Using Makefiles in Python projects

[Thomas Jollans]

On 11/11/2019 14:23, Rhodri James wrote:

On 09/11/2019 23:50, Thomas Jollans wrote:

On 09/11/2019 21:30, Chris Angelico wrote:

On Sun, Nov 10, 2019 at 2:10 AM Thomas Jollans  wrote:

On 07/11/2019 20:20, Vitaly Potyarkin wrote:

What do you think of using Makefiles for automating common chores in
Python projects? Like linting, type checking and testing?

I've come up with a reusable Makefile for automating virtual 
environment
management in Python projects. I think it can be useful for 
simplifying
the onboarding of new developers (both new to project and new to 
Python)

and for documenting project's development practices.

Here it is:
- Repo: https://github.com/sio/Makefile.venv
- Demo screencast: https://asciinema.org/a/279646

What do you think? Is this useful or I'm just unaware of some tool 
that

abstracts venv chores away better?


As others have said, make is a useful tool and many people use it for
different purposes in their Python projects. Nothing wrong with that.

HOWEVER, at risk of stating the obvious, using Makefiles written 
for/on
*nix systems on Windows is a bit of a hassle. If, for some reason, 
your

software is *nix-only anyway, that's fine. If not, using make means
sacrificing some portability.

If your software runs on Windows, of you think it might run on Windows
in the future, maybe consider writing simple Python scripts for
platform-independent tasks rather than makefiles and shell scripts.


Are you assuming that every Windows system has Python, but that you
can't get make or bash? Because neither half of that is true. I've
happily used makefiles on Windows, and these days, bash is as easy to
get hold of as Python is.

ChrisA


That's why I say "a bit of a hassle". You can get a MSYS set up 
(whether from Git for Windows or otherwise). You can get it to play 
nice with the right Python installation and the Python scripts you 
presumably want to call from the Makefile. But all of that is a bit 
of a hassle.


If you've got almost any development environment for Windows, you've 
got a version of make.  I quite like the NMake that comes with Visual 
Studio, for example, and use it in preference to the IDE when I can. 
Yes, it's a hassle, but it's a hassle you're going to go through anyway.


I'm sure it's possible to write Makefiles that work with both GNU make 
and NMake, but I imagine it's a rather limiting and thankless enterprise.


Is that something you actually do? (Maybe it's great, I really wouldn't 
know. Do tell!)



--
https://mail.python.org/mailman/listinfo/python-list

Re: Using Makefiles in Python projects

[Thomas Jollans]

On 09/11/2019 21:30, Chris Angelico wrote:

On Sun, Nov 10, 2019 at 2:10 AM Thomas Jollans  wrote:

On 07/11/2019 20:20, Vitaly Potyarkin wrote:

What do you think of using Makefiles for automating common chores in
Python projects? Like linting, type checking and testing?

I've come up with a reusable Makefile for automating virtual environment
management in Python projects. I think it can be useful for simplifying
the onboarding of new developers (both new to project and new to Python)
and for documenting project's development practices.

Here it is:
- Repo: https://github.com/sio/Makefile.venv
- Demo screencast: https://asciinema.org/a/279646

What do you think? Is this useful or I'm just unaware of some tool that
abstracts venv chores away better?


As others have said, make is a useful tool and many people use it for
different purposes in their Python projects. Nothing wrong with that.

HOWEVER, at risk of stating the obvious, using Makefiles written for/on
*nix systems on Windows is a bit of a hassle. If, for some reason, your
software is *nix-only anyway, that's fine. If not, using make means
sacrificing some portability.

If your software runs on Windows, of you think it might run on Windows
in the future, maybe consider writing simple Python scripts for
platform-independent tasks rather than makefiles and shell scripts.


Are you assuming that every Windows system has Python, but that you
can't get make or bash? Because neither half of that is true. I've
happily used makefiles on Windows, and these days, bash is as easy to
get hold of as Python is.

ChrisA


That's why I say "a bit of a hassle". You can get a MSYS set up (whether 
from Git for Windows or otherwise). You can get it to play nice with the 
right Python installation and the Python scripts you presumably want to 
call from the Makefile. But all of that is a bit of a hassle.



--
https://mail.python.org/mailman/listinfo/python-list

Re: Using Makefiles in Python projects

[Thomas Jollans]

On 07/11/2019 20:20, Vitaly Potyarkin wrote:

What do you think of using Makefiles for automating common chores in
Python projects? Like linting, type checking and testing?

I've come up with a reusable Makefile for automating virtual environment
management in Python projects. I think it can be useful for simplifying
the onboarding of new developers (both new to project and new to Python)
and for documenting project's development practices.

Here it is:
- Repo: https://github.com/sio/Makefile.venv
- Demo screencast: https://asciinema.org/a/279646

What do you think? Is this useful or I'm just unaware of some tool that
abstracts venv chores away better?

As others have said, make is a useful tool and many people use it for 
different purposes in their Python projects. Nothing wrong with that.


HOWEVER, at risk of stating the obvious, using Makefiles written for/on 
*nix systems on Windows is a bit of a hassle. If, for some reason, your 
software is *nix-only anyway, that's fine. If not, using make means 
sacrificing some portability.


If your software runs on Windows, of you think it might run on Windows 
in the future, maybe consider writing simple Python scripts for 
platform-independent tasks rather than makefiles and shell scripts.



-- Thomas


--
https://mail.python.org/mailman/listinfo/python-list

Re: return a ctypes object to C

[Thomas Jollans]

On 31/10/2019 14.13, Arnaud Loonstra wrote:
> On 30-10-2019 09:32, Arnaud Loonstra wrote:
>> Hi all,
>>
>> I'm trying to wrap my head around the ctypes API. I have a C
>> structure I wish to create in Python and then return from python to C.
>>
>> So a python method is called from C and needs to return an object
>> which we then process in C again.
>>
>> I have a binding to access and create the C methods and structures so
>> in Python I can call the Zmsg() constructor. I now need to return this.
>>
>> My python test method is simply:
>>
>> def actor_test( *args, **kwargs):
>>  print("test")
>>  msg = Zmsg()
>>  frame = Zframe(b"Hello", 5)
>>  msg.prepend(frame)
>>  return msg
>>
>> the method is called from C as follows:
>>
>> PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
>>
>> This correctly calls the method. However the returned object is of
>> course a PyObject*. The debugger says it's
>>
>> ""    PyObject
>>  [class]    ""
>>  [super class]    ""
>>  [meta type]    ""
>>  ob_refcnt    1    Py_ssize_t
>>
>> However how I can I get it back to the original C type (zmsg_t *)
>>
>> Any help really appreciated.
>>
>
> What I've found so far is that I can return the address of the ctypes
> object.
>
> msg = Zmsg()
> frame = Zframe(b"Hello", 5)
> msg.prepend(frame)
> return addressof(msg._as_parameter_.contents)
>
> In C I can then cast it back to the original type.
>
> PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
> assert(pReturn);
> long bla = PyLong_AsLong(pReturn);
> zmsg_t* test = (zmsg_t *)bla;
> assert(test);
> char *hello = zmsg_popstr(test);
> assert(hello);
> assert(streq(hello, "Hello"));
>
> This works, I'm not sure if this is the right way. It also creates a
> complicated setup with the garbage collector.
>
> Anybody better ideas?

You've already got a complicated setup with your ctypes objects...

If you're using the Python C API anyway, why would you use ctypes at
all? You can create custom Python types in C to wrap your C pointers.

Alternatively: if you're using ctypes anyway, why use the Python C API
at all? You can create C function pointers from Python functions with
ctypes.

If you're mixing two different ways of interfacing Python and C, the
result will ALWAYS be messy. Better to stick to one. Personally, I
prefer cffi or cython depending on the situation, as I find them clearer
and easier to use than ctypes. Using the Python C API directly is
usually the hardest to understand and the easiest to get wrong.

-- Thomas



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert a scientific notation to decimal number, and still keeping the data format as float64

[Thomas Jollans]

On 18/10/2019 10:35, doganad...@gmail.com wrote:

Here is my question:


I am using the numpy.std formula to calculate the standart deviation. However, 
the result comes as a number in scientific notation.
Therefore I am asking, How to convert a scientific notation to decimal number, 
and still keep the data format as float64 ?

Or is there any workaround to get the initial standart deviation result as a 
decimal number?


Here is my code:

stdev=numpy.std(dataset)
print(stdev)
 Result: 4.449e-05


print(stdev.dtype)
 Result: float64


Solutions such as this:

stdev=format(stdev, '.10f')
converts the data into a string object! which I don't want.


Expected result: I am willing to have a result as a decimal number in a float64 
format.


Float64 is a binary number format, not a decimal format. What you ask 
can't be done.


As to what you want:

For most purposes, you can think of the float64 object storing the 
Platonic ideal of the number, not any particular representation of it. 
0.5 and 5E-5 are the same number. End of story.


If you want to work with the number, you don't care what it looks like. 
If you want to display the number, you do care what it looks like, and 
you want a str rather than a float.





System: (Python 3.7.4 running on Win10)


Regards,



--
https://mail.python.org/mailman/listinfo/python-list

Re: Generate simple image on a standalone Raspberrry Pi

[Thomas Jollans]

On 27/09/2019 14.43, Roy Hann wrote:
> I am designing a mobile application to run on a Raspberry Pi 3 model B.
> It will not have any Internet access. I need to generate a static image
> consisting of a simple arc representing (say) a speedometer or a
> pressure gauge. The image will need to be regenerated every 5 seconds.
> The image must be displayed in a web browser (served by gunicorn
> running on the Pi). I prefer it to be a PNG but that is only a
> preference.
>
> Plotly is amazing but big and slow. Pygal looked acceptable but
> Internet access is required to render the SVG. 

There's always matplotlib.

Depending on how simple it needs to be, pillow might do the trick.

>
> Has anyone any suggestion for a lightweight solution? 
>
> Roy

-- 
https://mail.python.org/mailman/listinfo/python-list

Fwd: numpy results in segmentation fault

[Thomas Jollans]

Please reply on-list. (both of you)



 Forwarded Message 
Subject:Re: numpy results in segmentation fault
Date:   Mon, 16 Sep 2019 17:04:57 +0530
From:   Test Bot 
To: Pradeep Patra 
CC: Thomas Jollans 



Firstly, in response to this
"
I tried to install numpy with 3.7.3 and it is for some
reason not working and after import when I run import numpy at python
console and press enter I get >>? i,e its not working properly.
"

the >> prompt after import numpy signifies that the numpy module has
been loaded and is available in the session.

Can you please provide the traceback you are getting along with the input.

PS - Also, you have some coins like thing on hackerrank I guess to
reveal the test cases, in case everything else fails.

On Mon, Sep 16, 2019 at 3:08 PM Pradeep Patra mailto:smilesonisa...@gmail.com>> wrote:

Yes it is crashing in the hackerrank site and the testcases fails with
segmentation fault. I tried to install numpy with 3.7.3 and it is
for some
reason not working and after import when I run import numpy at python
console and press enter I get >>? i,e its not working properly.

Can you please help letting me know the python and numpy compatibility
matrix or I am missing anything?

I tried some of the numpy code from the other github and it also
fails with
the segmentation fault :-(. I am guessing some numpy version compatility
issue or some environment issue.

    On Thu, Sep 12, 2019 at 8:00 PM Thomas Jollans mailto:t...@tjol.eu>> wrote:

> On 12/09/2019 15.53, Pradeep Patra wrote:
> > Hi ,
> >
> > I was trying to solve the hackerrank and was using python 3.7.x.
> > https://www.hackerrank.com/challenges/np-concatenate/problem
> >
> > While running the code sometimes I get success result and
sometimes it
> > fails with "Segmentation Fault" at Hacker rank UI. I dont have
any clue
> why
> > the code is crashing ? Does anyone have any idea?
>
>
> Are you sure it's your code that's crashing, and not something beyond
> your control? (Such as the software that is starting Python for you)
> Does it depend on the input? Can you reproduce the issue in a
controlled
> environment (i.e. on your own PC)?
>
>
> >
> > Regards
> > Pradeep
> >
> > import numpy
> >
> > n,m,p = map(int,input().split())
> > tgt_arr1 = []
> > for i in range(n):
> >     row = list(map(int,input().split()))
> >     tgt_arr1.append(row)
> > tgt_arr2 = []
> > for j in range(m):
> >     row = list(map(int,input().split()))
> >     tgt_arr2.append(row)
> >
> > num_arr1 = numpy.array(tgt_arr1,int)
> > num_arr2 = numpy.array(tgt_arr2,int)
> >
> > print(numpy.concatenate((num_arr1,num_arr2),axis=0))
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: numpy results in segmentation fault

[Thomas Jollans]

On 12/09/2019 15.53, Pradeep Patra wrote:
> Hi ,
>
> I was trying to solve the hackerrank and was using python 3.7.x.
> https://www.hackerrank.com/challenges/np-concatenate/problem
>
> While running the code sometimes I get success result and sometimes it
> fails with "Segmentation Fault" at Hacker rank UI. I dont have any clue why
> the code is crashing ? Does anyone have any idea?


Are you sure it's your code that's crashing, and not something beyond
your control? (Such as the software that is starting Python for you)
Does it depend on the input? Can you reproduce the issue in a controlled
environment (i.e. on your own PC)?


>
> Regards
> Pradeep
>
> import numpy
>
> n,m,p = map(int,input().split())
> tgt_arr1 = []
> for i in range(n):
> row = list(map(int,input().split()))
> tgt_arr1.append(row)
> tgt_arr2 = []
> for j in range(m):
> row = list(map(int,input().split()))
> tgt_arr2.append(row)
>
> num_arr1 = numpy.array(tgt_arr1,int)
> num_arr2 = numpy.array(tgt_arr2,int)
>
> print(numpy.concatenate((num_arr1,num_arr2),axis=0))


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: For the code to generate `zen of python'.

[Thomas Jollans]

On 07/08/2019 15.11, Hongyi Zhao wrote:
> Hi here,
>
> I noticed that the `zen of python' is generated by the following code:
>
> d = {}
> for c in (65, 97):
> for i in range(26):
> d[chr(i+c)] = chr((i+13) % 26 + c)
>
> print("".join([d.get(c, c) for c in s]))
>
>
> But the above code is not so easy for me to figure out.  Could someone 
> please give me some explanations on it?
>
> Regards


This code
[https://github.com/python/cpython/blob/master/Lib/this.py#L23] doesn’t
‘generate’ the Zen of Python, really. It just decrypts it. The Zen of
Python is hidden just above those lines.

Abj gur dhrfgvba vf: pna lbh svther bhg gur plcure?

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: how to generate a standard email form by default OS mail client?

[Thomas Jollans]

On 30/07/2019 10.06, dmitre...@gmail.com wrote:
> Hello,
>
> is Python capable of generating a email form (from standard data - address, 
> topic, string message) with raising it by a default OS email client, e.g. 
> Thunderbird? User would like to have a possibility to review/modify email 
> content inside the OS email client window before sending it.
>
> Raising gmail.com GUI form also could be a solution.
>
> Thank you in advance, D.

You could construct a mailto URI

https://en.wikipedia.org/wiki/Mailto

 subject = 'This is the subject'
>>> body = 'This is the text'
>>> to = 'g...@heaven.example.com'
>>> cc = 'friedrich.nietzs...@unibas.ch'
>> mailto_url = urllib.parse.quote(f'mailto:{to}?') +
urllib.parse.urlencode({'cc': cc, 'subject': subject, 'body': body},
quote_via=urllib.parse.quote)

If you open that URI with the webbrowser module, it should work. Well,
it might. On my system, Chrome refuses to open a mailto URI like this,
but Firefox plays along.

The better option would be to call the mail program directly (such as
using the subprocess module), but how you find out what to call will
depend on your OS. If this is just for one PC and you use Thunderbird,
then you might as well hard-code the Thunderbird executable, of course...



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .python_history file

[Thomas Jollans]

On 29/07/2019 23:06, Harry Grey wrote:
> Hy to Everyone
> first of all : Sorry for my english
> 
> what do you think about introducing a feature that allows you to group
> by date
> and after to filter the istruction that are written in the
> .python_history file:
> maybe with a new structure for .python_history: 
> 
> """
>   ###DATE-OF-SOME-DAY-START###
> 
> 
>   ###DATE-OF-SOME-DAY-END###
> 
> """
> 
> and maybe developing a new tool, or a core functionality of python
> interpreter callable as an argument in the cmd, that allow you to parse
> the .python_history file in given date:
> 
> With a tool:
>   
>   PhYstory -d 16 -m 03 -y 2019


Hi!

IMHO, this is a rather niche use case. And I suppose you could
relatively easily implement it yourself with a site hook that adds magic
comments to ~/.pyhistory as you suggest. If you want this functionality,
there's no need for it to be part of Python itself.

FWIW, IPython, as far as I can tell, already saves some timestamp
information in its history file.

-- Thomas

> 
> or like python Functionality:
> 
>   python -hist 16-03-2019/17-03-2019
> 
> and why not? why not have a the chance to enable and disable, at will,
> writing in the .python_history:
> 
> ""
>   >>> skip_hist
>   >>>
>   >>> # Make some unusefull instruction
>   >>> # Ignored in the history
>   >>> class TryClass:
>   >>> # Some Code
>   >>> 
>   >>>
>   >>> tc = TryClass()
>   >>> keep_hist
>   >>> # make some interesting code
>   >>> # and keep it in the history file.
> """
> 
> Thanks
> 
> DG.
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Definite or indefinite article for non-singletons?

[Thomas Jollans]

On 28/07/2019 17:13, MRAB wrote:
> [snip]
> 
> It's strange that "all the things" (meaning "all of the things") is OK,
> but otherwise it's "one of the things", "some of the things", etc.

Is it?

It's the same in French, Dutch and German. Can't tell if it just makes
sense or if it's Common Average European doing its thing.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3.7 - Reading text from Terminal window

[Thomas Jollans]

On 26/07/2019 06.33, nir.za...@gmail.com wrote:
> Hi,
>
> I want it to wait for a specific text on the terminal window that it opens 
> (putty.exe), how can i do it? 
>
> from pywinauto.application import Application
> from pynput.keyboard import Key, Controller
>  
> print('Configuring')
>  
> app = Application().start(cmd_line=u'"C:\\...putty.exe" ')
> puttyconfigbox = app.PuTTYConfigBox
> puttyconfigbox.wait('ready')
> listbox = puttyconfigbox.ListBox
> listbox.select(u'COM5')
> button = puttyconfigbox[u'&Load']
> button.click()
> button2 = puttyconfigbox[u'&Open']
> button2.click()
>  
> keyboard = Controller()
> keyboard.press(Key.enter)
>  
> keyboard.type('password1')
> keyboard.press(Key.enter)
> keyboard.type('enable\nconf t\n')
>
Not a clue, but why are you using PuTTY at all? Can't whatever it is
you're doing be achieved much more simply using e.g. pyserial, accessing
the COM port directly from Python?

-- T


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Thomas Jollans]

On 23/07/2019 04.27, DL Neil wrote:
> On 23/07/19 11:00 AM, Ethan Furman wrote:
>> On 07/20/2019 05:02 PM, DL Neil wrote:
>>
>>> Upon closer inspection, I realised it didn't just fail; it failed
>>> badly! Some silly, little, boy had imported the PythonEnvironment
>>> class but failed to ALSO import PythonVersionError. So, the reported
>>> error was not the expected exception!
>>
>> I don't understand the significance of not importing PythonVersionError:
>>
>> - PythonEnvironment does not need it to be imported
>>
>> - when PythonEnvironment raises PythonVersionError you still get
>> PythonVersionError
>>
>> - if your code says `except PythonVersionError` and you haven't
>> imported it you'll get a NameError
>>
Actually, if your code says ‘except PythonVersionError’ this will only
raise a NameError if there is an actual exception.

(this was not what I expected - I learned something new today!)

If no exception is being handled, the except statements are never
examined. This makes this harder to spot than you might think!

% cat test.py   
try:
    print('ok')
except FakeError:
    print('ah shit')

% python3 test.py
ok



That makes me think...


def get_exc(exc):
    print("we're checking for exception", exc)
    return exc


try:
    print("raising no. 1")
    raise ValueError
except get_exc(TypeError):
    print("t'was a TypeError")
except get_exc(ValueError):
    print("t'was a ValueError")

try:
    print("raising no. 2")
    raise TypeError
except get_exc(TypeError):
    print("t'was a TypeError")
except get_exc(ValueError):
    print("t'was a ValueError")


>> So, what's the issue?
>
>
> Have I correctly understood the question?
>
> NameError conveys nothing to the user.
> PythonVersionError is more communicative - and speaks volumes to 'us'.
>
> The mainline code is something like:
>
> p = PythonEnvironment()
> try:
>     p.compatibility( ...spec... )    # eg must be Py3 not 2.n
> except PythonVersionError:
>     print( more illuminating errmsg )
>
> If I am 'the user' I'd be quite happy without the try...except; but
> mere mortals need/deserve something more. Accordingly, the
> PythonVersionError custom exception/class.
>
> Yes, we could trap NameError, but that might also catch other
> name-errors (unlikely in this example, but 'just sayin'). Thus the
> greater specificity of the custom class.
>
>
> NB please see alternative 'solution' proposed (for critique) as
> "Nesting Custom Errors in Classes" discussion topic.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces: memory vs 'pollution'

[Thomas Jollans]

On 22/07/2019 07.06, DL Neil wrote:
>
> Current thoughts:
>
> import environment_module as em
>
> - so, even more of an abbreviation than suggested!?
> - I rarely need to write a long list of import statements, so there
> won't be many.
> - not normally using such abbreviations in my code, they will stand-out.
> - considered using upper-case, eg "EM" - it is a form of constant
> after-all


Just FYI, in the scientific Python community certain short abbreviations
are the norm. Many modules have a ‘standard’ abbreviation that most
people use, minimizing confusion.

import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt
import pandas as pd
import xarray as xr

and so on.

As long as you're consistent and use the same abbreviation across your
entire codebase, and you put the imports at the top where people can
find them, I think using 2–4 letter abbreviations, even without any
decoration, is a fine approach.

-- Thomas


> - considered adding a single under(-line) suffix, eg "em_" (on the
> basis of "made you think"). No, don't make me think (too much)!
> - so, perhaps a two-letter abbreviation with a single under(-line), eg
> "e_m", won't be confused with other naming conventions and yet
> stands-out. (sadly, that is "stands-out" to me, but 'everyone else'
> won't know its significance...)
>
> The try...except construct is a brilliant idea because it does exactly
> what I asked - requires both the class (what I wanted to include) AND
> the custom-exception class (what it needs included).
>
> If the class does not have any related error classes, then the
> try...except can simply check for 'exists?'...
>
>
> It's a habit I'm about to adopt. Many thanks! 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Thomas Jollans]

On 17/07/2019 09.58, Barry Scott wrote:
>
>> On 16 Jul 2019, at 20:48, Dan Stromberg  wrote:
>>
>>
>>  
>> A question arises though: Does threading.active_count() only show Python 
>> threads created with the threading module?  What about threads created with 
>> the thread module?
> Only pythons threads, if you think about it why would python care about 
> threads it does not control?


As the docs say, this counts threading.Thread objects. It does not count
all threads started from Python: threads started with the _thread
module, for instance, are not included.

What is more, threads started in external libraries can acquire the GIL
and run Python code. A great example of this are QThreads in a PyQt5
application: QThreads are started by the Qt runtime, which calls a Qt
slot. This Qt slot then might happen to be implemented in Python. I'm
sure other libraries do similar things.

Example with _thread just to check active_count behaviour:

#!/usr/bin/env python3

import threading
import _thread
import time

def thread_func(i):
    print('Starting thread', i)
    time.sleep(0.5)
    print('Thread done', i)

print('Using threading.Thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
time.sleep(0.1)
print('active threads:', threading.active_count())
t1.join()


print('Using threading & _thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
t2_id = _thread.start_new_thread(thread_func, (2,))
time.sleep(0.1)
print('active threads:', threading.active_count())
time.sleep(0.6)
print('Done, hopefully')



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Accumulate , Range and Zeros

[Thomas Jollans]

On 13/07/2019 11:54, Abdur-Rahmaan Janhangeer wrote:
> Greetings,
> 
> Given this snippet
> 
> from itertools import *
> import operator
> 
> 
> x = [1, 2, 3] # [0, 1, 2, 3, ..., 10]
> 
> y = accumulate(x, operator.mul)
> 
> print(list(y))
> 
> why does x = list(range(5)) produces only zeros?

What would you expect it to produce?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: super or not super?

[Thomas Jollans]

On 12/07/2019 16.12, Paulo da Silva wrote:
> Hi all!
>
> Is there any difference between using the base class name or super to
> call __init__ from base class?

There is, when multiple inheritance is involved. super() can call
different 'branches' of the inheritance tree if necessary.


Let me demonstrate:


class A1:
    def __init__(self):
    super().__init__()
    print('A1 called')

class B1(A1):
    def __init__(self):
    super().__init__()
    print('B1 called')

class C1(A1):
    def __init__(self):
    super().__init__()
    print('C1 called')

class D1(B1,C1):
    def __init__(self):
    super().__init__()
    print('D1 called')


class A2:
    def __init__(self):
    object.__init__(self)
    print('A2 called')

class B2(A2):
    def __init__(self):
    A2.__init__(self)
    print('B2 called')

class C2(A2):
    def __init__(self):
    A2.__init__(self)
    print('C2 called')

class D2(B2,C2):
    def __init__(self):
    super().__init__()
    print('D2 called')

if __name__ == '__main__':
    D1()
    print('---')
    D2()


##


% python3 super_demo.py
A1 called
C1 called
B1 called
D1 called
---
A2 called
B2 called
D2 called


>
> class C1:
>   def __init__(self):
>   ...
>
> class C2(C1):
>   def __init__(self):
>   C1.__init__(self) or super().__init__() ??
>   ...
>
> I have been using super, but I see some scripts where the base class
> name is used.

Just use super(), especially in __init__.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Matplotlib import image as float32

[Thomas Jollans]

On 01/07/2019 21:08, Markos wrote:
> Hi,
> 
> I observed that matplotlib reads an image file (PNG) as float32:
> 
> Please, how to read this file as int8 to get RGB in range of 0-255?

You may want to try a different library.

scikit-image's imread function will give you the image as an integer
array. This uses the imageio module under the hood.

OpenCV uses int8's for everything IIRC, but using that only makes sense
if you also want its image processing functionality

-- Thomas

> 
> Thank you,
> 
> Markos
> 
>> import numpy as np
> 
>> import matplotlib.pyplot as plt
> 
>> import matplotlib.image as mpimg
> 
>> imagem = mpimg.imread('lenna.png')
> 
>> print (imagem)
> 
> [[[0.8862745  0.5372549  0.49019608]
>   [0.8862745  0.5372549  0.49019608]
>   [0.8745098  0.5372549  0.52156866]
>   ...
> 
>> print (imagem.dtype)
> 
> float32
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing and memory management

[Thomas Jollans]

On 03/07/2019 18.37, Israel Brewster wrote:
> I have a script that benefits greatly from multiprocessing (it’s generating a 
> bunch of images from data). Of course, as expected each process uses a chunk 
> of memory, and the more processes there are, the more memory used. The amount 
> used per process can vary from around 3 GB (yes, gigabytes) to over 40 or 50 
> GB, depending on the amount of data being processed (usually closer to 10GB, 
> the 40/50 is fairly rare). This puts me in a position of needing to balance 
> the number of processes with memory usage, such that I maximize resource 
> utilization (running one process at a time would simply take WAY to long) 
> while not overloading RAM (which at best would slow things down due to swap). 
>
> Obviously this process will be run on a machine with lots of RAM, but as I 
> don’t know how large the datasets that will be fed to it are, I wanted to see 
> if I could build some intelligence into the program such that it doesn’t 
> overload the memory. A couple of approaches I thought of:
>
> 1) Determine the total amount of RAM in the machine (how?), assume an average 
> of 10GB per process, and only launch as many processes as calculated to fit. 
> Easy, but would run the risk of under-utilizing the processing capabilities 
> and taking longer to run if most of the processes were using significantly 
> less than 10GB
>
> 2) Somehow monitor the memory usage of the various processes, and if one 
> process needs a lot, pause the others until that one is complete. Of course, 
> I’m not sure if this is even possible.
>
> 3) Other approaches?
>

Are you familiar with Dask? 

I don't know it myself other than through hearsay, but I have a feeling
it may have a ready-to-go solution to your problem. You'd have to look
into dask in more detail than I have...


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Do I need a parser?

[Thomas Jollans]

On 01/07/2019 17:23, josé mariano wrote:
> Dear All,
> 
> Thank you very much for your valuable input.
> Thanks Alan for your kind words. I'm not Spanish, I'm Portuguese, but I know 
> what you mean.
> Thomas, I was able to track down the author but he is not willing to release 
> the source code. The executable is free but apparently the source is not. 
> 
> From your comments I came up with this general configuration for the new 
> software. Please tell me what you think.
> 
> The new software would use a settings files in one "standard" format. I like 
> INI. It's note very powerful, but is easy to read and enough for the matter 
> at hand. I could then use configparser to parse the settings to the main 
> module. One separate module would convert the original format into the new 
> one. 
> 
> The same for the script files. The new format would be plain python, in one 
> separated file, that could be imported into the main file. A separated module 
> would convert the old script format to the new one (python).
> 
> Both modules could be run explicitly or the main software would take as input 
> the original files and run the conversion modules.
> 
> In this way, old users could use their existing scripts and settings files 
> and new users that are willing to learn some basic python could use the new 
> format.
> 
> So, besides the new software, i would need to write one module to convert the 
> settings from the old format to the new one, maybe using the ideas of Thomas, 
> and another module to convert the sripts.
> 
> Is this a good approach. You comment are  very welcome. 


Sounds reasonable to me. Personally I prefer YAML (parsed e.g. with the
ruamel.yaml module) to INI because it's more flexible: you get (deeper)
nesting and data types (i.e. numbers and strings are different) - but
configparser is a fine module to use for config files. Some people like
using Python modules as config files.

Obviously, as Cameron points out, using Python instead of a custom
scripting language has security implications, that go with the increased
power that the user has. However, in a laboratory setting, where the
users are scientists who have physical access to the machine anyway, I
really don't think this is a concern. And the added flexibility may be
worth a lot down the line!

(I would say that though, as I suggested it)

Don't forget that many scientists frequently use Python anyway.

- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Plumbing behind super()

[Thomas Jollans]

On 30/06/2019 01:15, adam.pre...@gmail.com wrote:


Whoops. Now I need to figure out how the interpreter knows that change_a is a 
method and knows what self to feed it. I'm assuming that's in the cell 
variables similar to what super()'s doing as explained here. I haven't 
implemented cell variables so this is where I'm stuck in a sand pit.


Look up descriptors.

Actually, carefully (re)read all of 
https://docs.python.org/3/reference/datamodel.html


--
https://mail.python.org/mailman/listinfo/python-list

Re: Handle foreign character web input

[Thomas Jollans]

On 28/06/2019 22:25, Tobiah wrote:

A guy comes in and enters his last name as RÖnngren.

With a capital Ö in the middle? That's unusual.


So what did the browser really give me; is it encoded
in some way, like latin-1?  Does it depend on whether
the name was cut and pasted from a Word doc. etc?
Should I handle these internally as unicode?  Right
now my database tables are latin-1 and things seem
to usually work, but not always.



If your database is using latin-1, German and French names will work, 
but Croatian and Polish names often won't. Not to mention people using 
other writing systems.


So Günther and François are ok, but Bolesław turns into Boles?aw and 
don't even think about anybody called Владимир or محمد.





Also, what do people do when searching for a record.
Is there some way to get 'Ronngren' to match the other
possible foreign spellings? 


--
https://mail.python.org/mailman/listinfo/python-list

Re: Do I need a parser?

[Thomas Jollans]

On 29/06/2019 14:39, josé mariano wrote:

Dear all,

I'm sure that this subject has been addressed many times before on this forum, 
but my poor knowledge of English and of computer jargon and concepts results on 
not being able to find the answer i'm looking for when I search the forum.

So here is my problem: I have this open source project for the scientific 
community were i want to duplicate an old MS-DOS application written in 
Fortran. I don't have the source code. The idea is to re-write the software in 
Python. Originally, the old application would would need to input files: one 
config file, written with a specific format (see below) and a second one, the 
so-called scrip file, that defines the sequence of operations to be performed 
by the main software, also written in a specific format.


Is there any way you can get the source code? Can you track down the 
original author? Are there old backups?


That might eliminate the need for rewriting, and, in any case, would 
make it so much easier to be sure you're doing the right thing and not 
missing something.





To make the transition to the new application as painless as possible to the 
users, because most of them have their collection of scrips (and settings) 
developed over the years and are not willing to learn a new script language, I 
would like to make the new app 100% compatible with the old input files.


Obviously. Make sure you have tests, tests, and more tests. If there's 
documentation, use it, but don't trust it.


That's assuming there are loads of old scripts, that just continuing to 
use the old program is not an option. DOSbox to the rescue?


Another option might be to write a script that parses the old files and 
converts them to something more friendly to your infrastructure, such as 
YAML config files and Python scripts. This has two benefits:


(1) a human can easily check the result. If there are some 
incompatibilities, they'll be easier to spot. If the script misses 
something, the user can add it in.


(2) it might be easier to add new features later

It is a more complex and less user-friendly solution, though.




The operation of the new software would be like this: From the shell, run 
"my_new_software old_script_file.***". The new software would load the 
old_script, parse it (?), set the internal variables, load the script and run it.

So, to get to my questions:

- To load and read the config file I need a parser, right? Is their a parser 
library where we can define the syntax of the language to use? Are there better 
(meaning easier) ways to accomplish the same result?


You need to parse the file, obviously. Python is good for this sort of 
thing. str.split() and the re module are your friends. The format looks 
reasonably simple, so I'd just parse it into a simple data structure 
with the basic tools Python provides.



- For the interpretation of the script file, I don't have any clue how to 
this... One important thing, the script language admits some simple control 
flow statements like do-wile, again written using a specific sintax.


From the look of it, it's one instruction per line, and the first word 
of the line is, in some sense, a command? In this case, one way I can 
think of to run it would be an interpreter structured something like this:



class ScriptRunner:
 def __init__(self, config, script):
 self._variables = {}
 self._config = config
 self._script_lines = []
 for line in script.split('\n'):
 line = line.strip()
 if line.startswith('!'):
 #comment
 continue
 cmd, *args = line.split()
 self._script_lines.append((cmd.lower(), args))

 def run_script(self):
 self._script_iter = iter(self._script_lines)
 while True:
 try:
 cmd, args = next(self._script_iter)
 self.dispatch(cmd, args)
 except StopIteration:
 return

    def dispatch(self, cmd, args):
    method_name = f'_cmd_{cmd}'
    method = getattr(self, method_name)
    method(args)

    def _cmd_set(self, args):
    varname, value = args
    self._variables[varname] = value

    def _cmd_while(self, loop_args):
    # check condition or something
    loop_body = []
    while True:
 try:
 cmd, args = next(self._script_iter)
 # MAGIC
 if cmd == 'endwhile':
 break
 else:
 loop_body.append((cmd, args))
 except StopIteration:
 raise RuntimeError('loop not ended')
    while condition_is_met:
    for cmd, args in loop_body:
 # otherwise, just carry on executing the loop body
 self.dispatch(cmd, args)


In any case, there are three things you need to keep track of: 
variables, where in the script you are, and what control 

Re: Plumbing behind super()

[Thomas Jollans]

On 28/06/2019 02:13, adam.pre...@gmail.com wrote:

I'm trying to mimick Python 3.6 as a .NET science project and have started to 
get into subclassing. The super() not-a-keyword-honestly-guys has tripped me 
up. I have to admit that I've professionally been doing a ton Python 2.7, so 
I'm not good on my Python 3.6 trivia yet. I think I have the general gist of 
this, but want some affirmation.

If you use super() in a method, all it does is load super as a global on to the 
interpreter stack and call it without any arguments. So I'm left to wonder how 
it's able to figure anything out when it's being literally given nothing...


Python is open source, so let's open the source!

The magic happens when super() is constructed, in super_init 
https://github.com/python/cpython/blob/a8b27e623d75377aabe50df27e97cab4e81a174a/Objects/typeobject.c#L7814


Once super_getattro is called, the super object has already been set up, 
and it already knows which object/class to look at.


As you can see, super_init takes the calling frame and checks the first 
argument of the previous call, which, when called from a method, will 
generally be 'self'


https://github.com/python/cpython/blob/a8b27e623d75377aabe50df27e97cab4e81a174a/Objects/typeobject.c#L7849

There's some additional code that, I think, handles being called from a 
closure (but I don't know the C api well enough to be sure)


We can sort-of emulate this in pure Python with the inspect module

###

def not_quite_super():
    f = inspect.stack()[1].frame
    obj = list(f.f_locals.values())[0]
    print(type(obj), obj)

class A:
    def __init__(self, a=0, b=1):
    c = a + b # this is just to add some more locals to the mix
    not_quite_super()

A()


###

Once you have the caller object, the rest is just like calling super() 
with arguments.


I'm not actually sure that Python example above is guaranteed to work, 
but I *think* it should in versions where dicts are ordered (3.6+). 
Obviously it doesn't handle all the cases super() handles.





except that it's not being given literally nothing:

static PyObject *
super_getattro(PyObject *self, PyObject *name)

I was thinking maybe self has become more special in Python 3.6, but I don't think that's 
true since I've ported code to Python3 before that had inner classes where I'd use 
"inner_self" to disambiguate with the outer self. And though I thought it was 
so at first, it just turned out I screwed up my little code snippet to expose it. If self 
was special then I presume I could find it in my lookups and inject it.

So how do I go from CALL_FUNCTION on a super() global without anything else on 
the stack to somehow having all the information I need? Is there something 
tracking that I'm in an object scope when calling stuff?



--
https://mail.python.org/mailman/listinfo/python-list

Re: pip vs python -m pip?

[Thomas Jollans]

On 21/06/2019 15.27, Malcolm Greene wrote:
> 64-bit Python 3.6.8 running on Windows with a virtual environment activated.
>
> "pip -v" reports 19.0.3
> "python -m pip" reports 19.1.1
>
> Is this behavior by design or a bug?

If the pip and python executables you're calling both live in the
virtual environment, then it might be a bug

> My takeaway is that its better to run "python -m pip ..." vs "pip ..." when 
> running pip related tasks.

It's a good rule of thumb. When you have multiple versions of Python
installed side-by-side, there can be a danger that you call the wrong
'pip' by accident.


I don't think this should be a concern in a virtual environment though

Thomas

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: python 2 to 3 conversion

[Thomas Jollans]

On 17/06/2019 15.14, Igor Korot wrote:
> Hi,
> Is there a place where there is a full list of incompatibilities between
> python 2 and python 3 is available and how to fix them?

‘What’s new in Python 3.0’ is a good starting point

https://docs.python.org/3/whatsnew/3.0.html

It doesn’t list all standard library changes, but it has the most
important ones. Obviously it can't include anything that third-party
modules do.

The main incompatibility is obviously strings, and there various modules
have adopted different strategies for the transition.


>
> I'm looking for a way to fix following code, which runs perfectly with python 
> 2
> (and make it work with both python 2 and python 3):
>
> if bytes[0:16].tostring() != '':

No idea what this is supposed to do as you didn't say what ‘bytes’ was.
I could imagine that the tostring() method returns a bytes rather than a
str…

Also, ‘bytes’ is the name of a built-in type in Python3. You don't want
to use that as a variable name any more! (You can, though)

> I know there are automated tools that can help you do the upgrade, but
> automated tools can do only so much

By many accounts, 2to3 doesn't work very well in the real world, at
least not any more.


>
> And I am not sure if they can just add python 3 code and include version 
> check.
>
> Thank you.



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How control a GUI for an unrelated application from a Python script?

[Thomas Jollans]

On 14/06/2019 01.49, Christian Seberino wrote:
> I have a third party GUI that manages some hardware.
>
> I want to control the hardware from a Python script.

Forget about the GUI, see if you can control your device without it.

See how well the device is documented. Maybe there's an API? If not for
Python, maybe for C? If not, maybe it's easy to write your own driver in
Python?

Writing a driver for a simple message-based wire protocol is not very
hard, and may be easier than reliably controlling somebody else's GUI
with a script.


>
> This seems to mean I need to somehow have Python code
>   that imitates a human doing the necessary
> actions on the GUI (selecting menu options, pressing buttons, etc.)
>
> Is this possible / easy / doable?
>
> Thanks!
>
> Chris

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Questions about the IO modules and C-api

[Thomas Jollans]

On 03/06/2019 04:02, Windson Yang wrote:
> I have some questions about the IO modules.
> 
> 1. My script:
> 
> f = open('myfile, 'a+b')
> f.close()
> 
> I added a printf statement at the beginning of _io_open_impl
> ,
> the output is:

Is this the output of running `python your_script.py`? If it is, it may
include calls to _io.open made during Python startup. Maybe you can add
a line to print out which file is being opened each time as well (this
is a bit tricky). Or you could add print() calls at the start and end of
your script to see what's happening during, what before, and what after
your script.

> 2. I'm not familiar with the C, How the c-api like
> PyObject_CallMethodObjArgs(self->raw,
> _PyIO_str_write, memobj, NULL)
> 
>  works?

If you haven't yet, I suggest you read the Python C API tutorial [1] and
consult the C API reference [2].

FWIW, the write call is ultimately here [3] via here [4].

[1] https://docs.python.org/3/extending/index.html
[2] https://docs.python.org/3/c-api/index.html
[3]
https://github.com/python/cpython/blob/b82e17e626f7b1cd98aada0b1ebb65cb9f8fb184/Python/fileutils.c#L1586
[4] https://github.com/python/cpython/blob/master/Modules/_io/fileio.c#L854

> 
> I guess this function will finally call the `write()` system call but I
> don't know why it would work. I found `self->raw` is an empty PyObject and
> `_PyIO_str_write` is a global variable which is NULL.  Why an empty
> PyObject have a write method? Why we didn't just use `write()` system call
> directly?

Oh, but it's not NULL, is it?

https://github.com/python/cpython/blob/331a6a56e9a9c72f3e4605987fabdaec72677702/Modules/_io/_iomodule.c#L761

I don't believe self->raw is "empty" either. It's initialized by
_io.open, isn't it?

As for why go through the python method rather than calling something in
C directly, that would be in order to allow BufferedWriter to be used
with different types of IO classes, not just files.


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why Python has no equivalent of JDBC of Java?

[Thomas Jollans]

On 19/05/2019 14.27, Marco Sulla via Python-list wrote:
> I programmed in Python 2 and 3 for many years, and I find it a fantastic
> language.
> 
> Now I'm programming in Java by m ore than 2 years, and even if I found its
> code much more boilerplate, I admit that JDBC is fantastic.

Python has a the "Python Database API" (DB API 2.0)
https://www.python.org/dev/peps/pep-0249/

All SQL database modules that I know of use this API. It's more
decentralized than JDBC (not that I know much about JDBC), but it does
the trick: as long as the SQL syntax is sufficiently compatible, you can
easily swap in one database for another.

> 
> One example over all: Oracle. If you want to access an Oracle DB from
> Python, you have to:

If Oracle wants to be a pain in the arse, that's Oracle's doing. The
open source databases aren't like that.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using a Variable Inside the String

[Thomas Jollans]

On 07/05/2019 18.43, Rob Gaddi wrote:
> This over here is my friend Bob.  What's 31 + 18 + Bob?

That's Numberwang!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Building a statically linked Python, and pip

[Thomas Jollans]

On 08/05/2019 03:18, Simon Michnowicz via Python-list wrote:
> Dear Group,
> I need to build a statically linked Python that has pip.

What a curious thing to need.

> I built a version following the instructions at
> https://wiki.python.org/moin/BuildStatically
> but pip was not present in the binary directory afterwards.
> 
> [...]
>
> ModuleNotFoundError: No module named '_queue'
> 

As the wiki instructions point out, if you want standard library modules
written in C (such as _queue), you have to set them up to be built
inside your binary.

So you'll have to figure out which modules you need for what you're
doing and how to include them in your build.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PYTHON equivalents of BITAND and BITSHIFT of MATLAB

[Thomas Jollans]

On 02/05/2019 06:47, blmadha...@gmail.com wrote:
> Hello Brian,
> 
> Thanks for your suggestion. Which is correct for MATLAB command: typeBits = 
> FCF << -9?
> 
> typeBits = FCF >> 9 
> 
> or 
> 
> typeBits = FCF >> -9
> 
> I mean to ask if it should be -9 or +9?

Why don't you just, you know, try them out?

Fire up MATLAB or GNU Octave (here I'm using MATLAB R2016a)

>> bitshift(1024, -9)

ans =

 2

>> bitshift(1024, +9)

ans =

  524288


And fire up Python


Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1024 >> 9
2
>>> 1024 << 9
524288
>>> 1024 << -9
Traceback (most recent call last):
  File "", line 1, in 
ValueError: negative shift count
>>>



> 
> Thanks in advance
> Madhavan
> 
> On Wednesday, May 1, 2019 at 11:22:10 PM UTC+5:30, Brian Oney wrote:
>> On Wed, 2019-05-01 at 10:35 -0700, blmadha...@gmail.com wrote:
>>> Hi,
>>>
>>> I have the following line from a MATLAB program with FCF (format: UInt_16) 
>>> as input:
>>>
>>> ftype = bitand(FCF, 7)
>>> typeBits = bitshift(FCF, -9)
>>> subtype = bitand(typeBits, 7)
>>>
>>> I wrote the following in Python for the above commands:
>>>
>>> ftype = FCF & 7
>>> typeBits = FCF << -9 --> Is this correct or FCF >> -9?
>>> subtype = typeBits & 7
>>>  
>>> Can someone help me write the equivalent command in PYTHON?
>>>
>>> Look forward to your suggestions.
>>
>> >From the Matlab doc:
>> '
>> intout = bitshift(A,k)
>> intout = bitshift(A,k,assumedtype)
>> Description
>>
>> example
>>
>> intout = bitshift(A,k) returns A shifted to the left by k bits,
>> equivalent to multiplying by 2k. Negative values of k correspond to
>> shifting bits right or dividing by 2|k| and rounding to the nearest
>> integer towards negative infinity. Any overflow bits are truncated. '
>>
>> So the equivalent would be:
>>
> typeBits = FCF >> 9
>>
>> Cheers
>> Brian
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: EmailMessage and RFC 2047

[Thomas Jollans]

On 30/04/2019 13.11, Peter J. Holzer wrote:
> Hi,
> 
> https://docs.python.org/3.7/library/email.header.html states:
> 
> | This module is part of the legacy (Compat32) email API. In the current
> | API encoding and decoding of headers is handled transparently by the
> | dictionary-like API of the EmailMessage class.
> 
> I understood this to mean that an EmailMessage does decode RFC 2047
> encoded header fields automatically.

I have no idea why it's doing what it's doing. The lesson appears to be:
don't use the compat32 mode unless you have to support Python 3.2 (which
you don't, because that's ridiculous)

>>> msg
'Subject: =?utf-8?q?=C3=89lys=C3=A9e?=\nContent-Type: text/plain;
charset="utf-8"\nContent-Transfer-Encoding: 7bit\nMIME-Version:
1.0\n\nTEST MESSAGE\n'
>>> p = email.parser.Parser(email.message.EmailMessage)
>>> p.parsestr(msg)['Subject']
'=?utf-8?q?=C3=89lys=C3=A9e?='
>>> p = email.parser.Parser(email.message.EmailMessage,
policy=email.policy.default)
>>> p.parsestr(msg)['Subject']
'Élysée'
>>>

Simple. Silly. Simply silly.

I've been confused by the unicode handling of this module before... [1]

— Thomas

[1]. https://mail.python.org/pipermail/python-list/2018-October/737726.html
[I'm still confused]

> 
> However this does not seem to be the case:
> 
> portia:~/tmp 13:06 :-) 16% python3
> Python 3.7.2+ (default, Feb 27 2019, 15:41:59) 
> [GCC 8.2.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 import email.parser
 import email.message
 p = email.parser.Parser(email.message.EmailMessage)
 f = open("test.msg", "r")
 m = p.parse(f)
 m
> 
 m["Subject"]
> '[luga] 
> =?utf-8?Q?=C3=84ndern_des_Passwortes_mittels_We?=\n\t=?utf-8?Q?bformular?='
 m["From"]
> 'Martin =?iso-8859-15?Q?W=FCrtele?= '

> 
> Am I using it wrong or did I misunderstand what is meant by "handled
> transparently by the dictionary-like API of the EmailMessage class"? If
> the latter, what does it mean?
> 
> hp
> 
> 



signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Most "pythonic" syntax to use for an API client library

[Thomas Jollans]

On 30/04/2019 09.52, Peter Otten wrote:
> Thomas Jollans wrote:
> 
>> On 29/04/2019 09.18, Peter Otten wrote:
>>> Jonathan Leroy - Inikup via Python-list wrote:
>>> alice.name = "Bob"  # __setattr__
>>>
>>> del customers[42]  # __delitem__
>>
>> do you want this sort of thing to update the upstream database directly?
> 
> I thought so. However, you have to answer this question no matter which of 
> the suggested APIs you choose.

True, but with a .update(key=value, ...) method (which takes multiple
kwargs) you don't have the potential issue of more requests than
necessary if you're updating multiple fields.

> 
>> Maybe there should be a commit() method on every object. Maybe not.
> 
> I originally had
> 
> db = api.connect(...)
> customers = db.customers
> 
> but then decided this was out of scope for the question. It would however 
> allow
> 
> with db:
> ...  # modify db
> # implicit commit, or rollback on exception
> 
>> I imagine it would be best if the data is cached locally (i.e. alice =
>> customers[1] does a query, print(alice.name) does not). In this case you
>> probably want the local/database separation to be consistent for both
>> getting and setting things.
> 
> That looks like a lot of work, with the potential to introduce additional 
> inconsistencies. But there's probably prior art.
> 
> How do ORMs like SQLObject handle this?

It's been a while, but I'm pretty sure the Django ORM at least has
caching everywhere.

>  
>>> del customers[alice]
>>
>> Are you sure about this?
> 
> On a scale from one to ten? Yes ;)
> 
>>>> #3 seems to be more "pretty" to me, but I did not find any "official"
>>>> recommendation online.
>>
>> I'd like #3 if it had square brackets after .customers. Of the
>> suggestions as they are I prefer #2.
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Most "pythonic" syntax to use for an API client library

[Thomas Jollans]

On 29/04/2019 09.18, Peter Otten wrote:
> Jonathan Leroy - Inikup via Python-list wrote:
> 
>> Hi all,
>>
>> I'm writing a client library for a REST API. The API endpoints looks like
>> this: /customers
>> /customers/1
>> /customers/1/update
>> /customers/1/delete
>>
>> Which of the following syntax do you expect an API client library to
>> use, and why?
>>
>> 1/
>> api.customers_list()
>> api.customers_info(1)
>> api.customers_update(1, name='Bob')
>> api.customers_delete(1)
>>
>> 2/
>> api.customers.list()
>> api.customers.info(1)
>> api.customers.update(1, name='Bob')
>> api.customers.delete(1)
>>
>> 3/
>> api.customers.list()
>> api.customers(1).info()
>> api.customers(1).update(name='Bob')
>> api.customers(1).delete()
>>
>> ...any other?
> 
> How about mimicking (to some extent) an existing interface, like a list, 
> dict, or set:
> 
> customers = api.customers
> 
> list(customers)  # __iter__
> 
> alice = customers[1]  # __getitem__
> 
> print(alice)  # __str__

This was my first thought seeing the third option as well, but...

> 
> alice.name = "Bob"  # __setattr__
> 
> del customers[42]  # __delitem__

do you want this sort of thing to update the upstream database directly?
Maybe there should be a commit() method on every object. Maybe not.

I imagine it would be best if the data is cached locally (i.e. alice =
customers[1] does a query, print(alice.name) does not). In this case you
probably want the local/database separation to be consistent for both
getting and setting things.

> del customers[alice]

Are you sure about this?

> 
>> #3 seems to be more "pretty" to me, but I did not find any "official"
>> recommendation online.

I'd like #3 if it had square brackets after .customers. Of the
suggestions as they are I prefer #2.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A request for examples of successful abstracts to scientific python conferences

[Thomas Jollans]

On 03/04/2019 18:18, Manolo Martínez wrote:
> Dear all,
> 
> I am thinking of submitting an abstract to EuroSciPy 2019, but (although
> I am an academic), I am not familiar with the conventions, do's and
> don'ts of submissions to CS conferences, and this one in particular.
> Would any kind reader of this list be willing to share with me
> (off-list) successful abstracts submitted to other editions of this or
> similar conferences? Also, perhaps, general resources that I should read
> before responding to a computer science CFP? That would be extremely
> helpful.

You can find the program of previous editions (including of course the
abstracts of all talks that were scheduled) on the EuroSciPy website.

> 
> Thanks in advance. Best,
> 
> Manolo Martínez
> 
> Universitat de Barcelona
> 
> https://manolomartinez.net
> 
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Thomas Jollans]

On 29/03/2019 12.39, Tony van der Hoff wrote:
> On 29/03/2019 11:08, Chris Angelico wrote:
>> On Fri, Mar 29, 2019 at 9:12 PM Tony van der Hoff  
>> wrote:
>>>
>>> Hello Chris.
>>> Thanks for your interest.
>>>
>>> On 28/03/2019 18:04, Chris Angelico wrote:
 On Fri, Mar 29, 2019 at 4:10 AM Tony van der Hoff  
 wrote:
>
> This'll probably work:

 You have a python3 shebang, but are you definitely running this under 
 Python 3?

>>> Absolutely.
>>>
 Here's a much more minimal example. Can you see if this also fails for you?

 import sys
 from jinja2 import Template
 print(Template("French: {{french}}").render({"french": "année"}))
 print(sys.version)

>>>
>>> Presumably you expect to run this from the command line. It works as
>>> expected:
>>>
>>> French: année
>>> 3.5.3 (default, Sep 27 2018, 17:25:39)
>>> [GCC 6.3.0 20170516]
>>>
>>> However, with a slight modification:
>>>
>>> #!/usr/bin/env python3
>>>
>>> import sys
>>> from jinja2 import Template
>>> print ("Content-type: text/html\n\n")
>>
>> Try: text/html; charset=utf-8
>>
> No difference
> 
>> That might be all you need to make the browser understand it
>> correctly. Otherwise, as Thomas says, you will need to figure out
>> where the traceback is, which can probably be answered by figuring out
>> what "running it in a browser" actually means.
>>
> 
> Running in browser:
> http://localhost/~tony/private/home/learning/jinja/minimal/minimal.py
> 
> In apache2.access.log:

So it's running in apache!

Now the question is what apache is doing. Is it running it as a CGI
script? Is it doing something clever for Python files (maybe involving
Python 2?)

... wild guess: if the script is running as CGI in an enviroment with an
ASCII-using "C" locale, with Python 3.5, you wouldn't be able to print
non-ASCII characters by default. I think. In any case I remember reading
about this problem (if this is the problem) being fixed in a newer
version of Python.

> ::1 - tony [29/Mar/2019:11:22:13 +] "GET
> /~tony/private/home/learning/jinja/minimal/minimal.py HTTP/1.1" 200 204
> "http://localhost/~tony/private/home/learning/jinja/minimal/";
> "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/72.0.3626.81 Safari/537.36"
> ::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"
> ::1 - - [29/Mar/2019:11:23:04 +] "-" 408 0 "-" "-"
> 
> So, 408 is a bit unusual for localhost. With the accented character
> removed, no timeout is reported. Maybe a clue.
> 
> Can find no other traceback. Nothing relevant in apache2/error.log
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Jinja and non-ASCII characters (was Re: Prepare accented characters for HTML)

[Thomas Jollans]

On 29/03/2019 11.10, Tony van der Hoff wrote:
> and running it in a browser (tried both chrome and Firefox),

How?

> it fails as before: blank web page.

No traceback? There must be a traceback somewhere. In a log file perhaps.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: uWISGI with Qt for Python

[Thomas Jollans]

On 13/03/2019 22:42, Israel Brewster wrote:
> 1) Is there a “better way”? This GitHub repo: 
> https://github.com/unbit/uwsgi-qtloop seems to indicate that it should be 
> possible to run a Qt event loop from within a uWSGI app, thus eliminating the 
> extra “subprocess” spinoff, but it hasn’t been updated in 5 years and I have 
> been unable to get it to work with my current Qt/Python/OS setup
> 
> 2) Baring any “better way”, is there a way to at least ensure that the 
> subprocess is killed in the event of parent death, or alternately to look for 
> and kill any such lingering processes on application startup?
> 
> P.S. The purpose of running the web server is to be able to load and use 
> Plotly charts in my app (via a QWebEngineView). So a “better way” may be 
> using a different plotting library that can essentially “cut out” the middle 
> man. I’ve tried Matplotlib, but I found its performance to be worse than 
> Plotly - given the size of my data sets, performance matters. Also I had some 
> glitches with it when using a lasso selector (plot going black). Still, with 
> some work, it may be an option.

In a similar situation (web component in a Qt app) we simply use
werkzeug's built-in web server in a Python/Qt thread. This works very
nicely, but the performance probably isn't perfect. Our use case is
allowing a small part of our experiments to be controlled from a phone
in the local network, so the performance requirements aren't severe at all.

As for live plotting in a GUI app, you're right, matplotlib isn't really
suitable. But pyQtgraph is: http://www.pyqtgraph.org/
I've had no problems with it at all for streaming live measurement data
to screen and interacting with reasonable-sized 2d images. I expect
that's your best option.

Despite what the description says, it does PyQt5 as well as PyQt4.

Best,
- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Convert Windows paths to Linux style paths

[Thomas Jollans]

On 12/03/2019 15.51, Malcolm Greene wrote:
> Looking for best practice technique for converting Windows style paths to 
> Linux paths. Is there an os function or pathlib method that I'm missing or is 
> it some combination of replacing Windows path separators with Linux path 
> separators plus some other platform specific logic?
> 
> Thank you,
> Malcolm
> 

Could you give some examples of what sort of paths you want to convert
in what way?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Configuring the REPL's tab completion

[Thomas Jollans]

On 10/03/2019 15.20, Chris Angelico wrote:
> I have absolutely no idea how to do this or even where to go looking,
> so I'd appreciate a starting pointer :)
> 
> When you're in the Python REPL (just the basic core one, not IDLE or
> anything), you can tab-complete global and built-in names, attributes
> of known objects, etc. But quoted strings work kinda weirdly - they
> try to tab-complete a global or keyword:
> 
> Python 3.8.0a0 (heads/master:8b9c33ea9c, Nov 20 2018, 02:18:50)
> [GCC 6.3.0 20170516] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 "in
> in  input(  int(
 "input("
> 'input('
> 
> I typed "in and hit tab twice, then typed p and hit tab, enter. It
> filled in the function name *input*, added an open parenthesis... and
> then closed the quote. Which doesn't make a lot of sense, but then,
> tab completing globals and keywords inside a text string doesn't make
> that much sense either.
> 
> What would be more useful would be tab-completing file names from the
> current directory.
> 
> open("Foo
> 
> to fill in the name of a file. Sure, not every quoted string is a file
> name (in fact, very few are), but I don't know of anything else that
> would feel natural. (Also, Pike's REPL behaves this way, so presumably
> it's of use to more people than me.)
> 
> Where would I start looking to try to make this happen? Doesn't
> necessarily have to be pushed upstream as a core Python feature; I'm
> guessing this can probably be done in sitecustomize.py. Anyone have
> tutorials on messing with tab completion? There's not a lot of info in
> the rlcompleter module docs.

I had a quick look at the code -

rlcompleter sets itself as the completer

using readline.set_completer


I imagine you could write your own completer, and call set_completer
again after rlcompleter has been imported.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Totally Legit Signing Key?

[Thomas Jollans]

On 04/03/2019 20:37, Peter Otten wrote:
> For once I tried to verify a download from python.org, following the steps 
> outlined at
> 
> https://www.python.org/downloads/#pubkeys
> 
> """
> You can import the release manager public keys by either downloading the 
> public key file from here and then running
> 
> gpg --import pubkeys.txt
> """
> 
> When I ran the command above I saw
> 
> $ gpg --import pubkeys.txt 
> gpg: Schlüssel 6F5E1540: "Ned Deily " 2 neue Signaturen
> gpg: Schlüssel 6A45C816: "Anthony Baxter " nicht 
> geändert
> gpg: Schlüssel 36580288: "Georg Brandl (Python release signing key) 
> " 2 neue Signaturen
> gpg: Schlüssel 7D9DC8D2: "Martin v. Löwis " nicht geändert
> gpg: Schlüssel 18ADD4FF: "Benjamin Peterson " 3 neue 
> Signaturen
> gpg: Schlüssel A4135B38: "Benjamin Peterson " 1 neue 
> Signatur
> gpg: Schlüssel A74B06BF: "Barry Warsaw " 138 neue Signaturen
> gpg: Schlüssel EA5BBD71: "Barry A. Warsaw " 6 neue Signaturen
> gpg: Schlüssel E6DF025C: "Ronald Oussoren " nicht 
> geändert
> gpg: Schlüssel F73C700D: "Larry Hastings " 2 neue 
> Signaturen
> gpg: Schlüssel AA65421D: "Ned Deily (Python release signing key) 
> " 1 neue User-ID
> gpg: Schlüssel AA65421D: "Ned Deily (Python release signing key) 
> " 20 neue Signaturen
> gpg: Schlüssel 487034E5: "Steve Dower (Python Release Signing) 
> " 8 neue Signaturen
> gpg: Schlüssel 10250568: Öffentlicher Schlüssel "Łukasz Langa (GPG langa.pl) 
> " importiert
> gpg: Schlüssel 487034E5: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> gpg: Schlüssel F73C700D: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> gpg: Schlüssel 6F5E1540: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> gpg: Schlüssel AA65421D: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> gpg: Schlüssel E6DF025C: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> gpg: Schlüssel EA5BBD71: Öffentlicher Schlüssel "Totally Legit Signing Key 
> " importiert
> [...]

Everything's working fine on your end. If you have a closer look, you'll
see that all of the "Totally Legit" keys have key IDs that are identical
to key IDs of actual Python release managers. e.g. in the last line,
EA5BBD71 refers to the key

pub   rsa1024 2015-05-22 [C]
  801BD5AE93D392E22DDC6C7AFEA3DC6DEA5BBD71
uid   [ unknown] Totally Legit Signing Key 

but it ALSO refers to the key

pub   dsa1024 2005-11-24 [SC]
  DBBF2EEBF925FAADCF1F3FFFD9866941EA5BBD71
uid   [ unknown] Barry A. Warsaw 
uid   [ unknown] Barry A. Warsaw 
uid   [ unknown] Barry A. Warsaw 
uid   [ unknown] Barry A. Warsaw 
uid   [ unknown] Barry Warsaw (GNU Mailman) 
uid   [ unknown] Barry A. Warsaw 
sub   elg2048 2005-11-24 [E]

The thing is that 32-bit key IDs are not secure and can easily be
cloned. [1]

I imagine that Barry at least knows this, seeing as he apparently cloned
his own old (compromised) key:

pub   rsa1024 2014-06-16 [SCEA] [revoked: 2016-08-16]
  2C7E264D238159CB07A3C350192720F7EA5BBD71
uid   [ revoked] Barry A. Warsaw 

What I imagine happened here is that whoever exported the pubkeys.txt
file did so on the basis of 32-bit key IDs. This is not ideal, as it
pulled in bogus keys, but there's no real harm done.

For good measure, I've put this on bpo (36191)

-- Thomas

[1] https://evil32.com/

> 
> Now "totally legit" does sound like anything but "totally legit". Is there a 
> problem with my machine, or python.org, or is this all "totally legit"?
> 
> Advice or pointers welcome.
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Version numbers in Standard Library

[Thomas Jollans]

On 01/03/2019 16:35, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] via Python-list wrote:
> Dear Python List,
> 
> A question. I help maintain a Python stack for users in my division here
> at NASA and one user asked about updating the re module to 2.4. I
> believe because he read the docs:
> 
> https://docs.python.org/2.7/library/re.html
> 
> where you see lines like "New in version 2.4" and he also did:
> 
> $ python2 -c 'import re; print (re.__version__)'
> 2.2.1
> 
> And, well, one can think "oh, a newer version is needed". I searched on
> conda, etc. and can't find it and finally realized that 2.4 meant Python
> 2.4, not re 2.4. (The 3.7 docs have lines like "Changed in version 3.7".)
> 
> My question to the pros here is what purpose do the __version__/version
> variables serve in the Python Standard Library?  I can understand in
> external packages, but once in the Standard Library...?
If a module that started life outside the standard library is included
in the standard library under the same name, it is taken aboard hook,
line and sinker. If it has a __version__, then that should probably
stay: code that used the module before it entered the standard library
might rely on it.

As you quite rightly point out, these lines are then completely
pointless, so there's no reason to ever change them.

> 
> For example, in re.py, that line was last changed 18 years ago according
> to git blame. In tarfile.py, the version string was last changed 12
> years ago. But in both, the modules were edited in 2018 so they haven't
> been static for a decade.
> 
> Are those strings there just for historic purposes?
> 
> Not a big deal, I was just wondering.
> 
> Thanks,
> Matt
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Thomas Jollans]

On 27/02/2019 22:39, Roel Schroeven wrote:
> Aren't we overthinking this?

No, it's just that nobody had found the evidence you did.

> 
> I think it's pretty clear that a variable is never deleted before it
> goes out of scope. A quick search in the documentation points me to
> (https://docs.python.org/3/reference/datamodel.html#objects-values-and-types):
> 
> 
> "Objects are never explicitly destroyed; however, when they become
> unreachable they may be garbage-collected. An implementation is allowed
> to postpone garbage collection or omit it altogether — it is a matter of
> implementation quality how garbage collection is implemented, *as long
> as no objects are collected that are still reachable*." (emphasis mine)
> 
> In the original example (without del), f is reachable everywhere in the
> function after the initial binding, so it can not be deleted.

Thanks!


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Thomas Jollans]

On 27/02/2019 16.41, Marko Rauhamaa wrote:
> Rhodri James :
>> The description of the with statement does explicitly say that the
>> context manager's __exit__() method won't be called until the suite
>> has been executed, so the reference to the open file must exist for at
>> least that long.
> 
> Yeah, but the *true* answer, of course, is:
> 
> def fun():
> f = os.open("lock", os.O_RDONLY)
> flock.flock(f, fcntl.LOCK_EX)
> do_stuff()
> sys.exit(0)
>
> Collect that!

No problemo!

try:
fun()
except SystemExit:
print("Oh no you don't!")

Collect this:

def fun():
f = os.open("lock", os.O_RDONLY)
flock.flock(f, fcntl.LOCK_EX)
do_stuff()
os.kill(os.getpid(), 9)


;-)


⸺ Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lifetime of a local reference

[Thomas Jollans]

On 27/02/2019 04.14, Alan Bawden wrote:
> Marko Rauhamaa  writes:
>> I couldn't find an immediate answer in the documentation.
> 
> I suspect that given the history of Python, pretty much everybody has
> always assumed that a Python implementation will not delete local variables
> early.  But I agree with you that the Python Language Reference does not
> appear to address this question anywhere!
> 

That's probably right. However, due to the nature of Python, things like
this are possible:

>>> v = 'a'
>>> def f():
... a, b, c = 1, 2, 3
... return eval(v)
...
>>> f()
1
>>> v = 'b'
>>> f()
2
>>>


What I mean to say is that it's not in general trivial (or even
possible) to tell if a local is referred to later in the function body.
I think we can say that a Python interpreter can't delete a local before
having executed any statement that could *possibly* access it.

If the inspect module's stack frame inspection machinery is supported,
then any function call might access any local... (though I don't think a
compliant Python implementation necessarily has to support the inspect
module fully). And we're back to where we started.

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quirk difference between classes and functions

[Thomas Jollans]

On 25/02/2019 21.15, Chris Angelico wrote:
> On Tue, Feb 26, 2019 at 6:58 AM DL Neil  
> wrote:
>>
>> On 26/02/19 5:25 AM, ast wrote:
>>> I noticed a quirk difference between classes and functions
>>>  >>> x=0
>>>  >>> class Test:
>>>  x = x+1
>>>  print(x)
>>>  x = x+1
>>>  print(x)
>> ...
>>
>>> Previous code doesn't generate any errors.
>>> x at the right of = in first "x = x+1" line is
>>> the global one (x=0), then x becomes local
>>> within a function, this is not allowed
>>>  >>> x = 0
>>>  >>> def f():
>>>  x = x+1
>>>  >>> f()
>>> UnboundLocalError: local variable 'x' referenced before assignment
>>> Since x is written inside the function, it is considered as a local
>>> variable and x in x+1 is undefined so this throw an exception
>>> Any comment ?
>>
>>
>> At first I misunderstood the question, and even now I'm slightly mystified:-
>>
>> Is the observation to do with the principles of "closure" (being applied
>> to the function) compared with the differences between class and
>> instance variables?
> 
> Classes and functions behave differently. Inside a function, a name is
> local if it's ever assigned to; but in a class, this is not the case.
> (Honestly, I'm not sure why this is, but it's hardly ever significant;
> metaprogramming seldom runs into this kind of thing.)

I imagine there's a justification for the difference in behaviour to do
with the fact that the body of a class is only ever executed once, while
the body of a function is executed multiple times. But I don't quite see it.




> 
>> A question from me: (I'm not an O-O 'native' having taken to it long
>> after first learning 'programming') I've used class attributes to hold
>> 'constants', eg
>>
>> MAXIMUM_WEIGHT = 100
>>
>> Thus only an assignment to which reference/assertions will be made 'later'.
>>
>> I have initialised a counter to zero and then incremented within the
>> instantiated object's __init__(), ie keeping track of how many of 'these
>> objects' have been instantiated.
> 
> Once you're inside a method, you would reference it with a dot -
> either "self.MAXIMUM_WEIGHT" or "cls.MAXIMUM_WEIGHT". So it's now an
> attribute, not a local name.
> 
>> So, I can imagine taking a value from outside the class' namespace,
>> modifying it in some way (per the first "x = x+1") and then retaining
>> the result as a class attribute (a calculation performed once - when the
>> class code is compiled).
> 
> Yes, this is perfectly legitimate. Not common but legit.
> 
>> However, (after that first calculation/definition of the class
>> attribute) why would one (normally) want to redefine the same class
>> attribute (the second x = x+1), at the 'class level'?
>> (perhaps just done for its amusement value?)
> 
> That would not be common either. If you're redefining a variable
> multiple times within a class block, you probably don't also have it
> at top level. (For instance, you can put a "for" loop inside a class
> statement to create attributes/methods, but the loop variable is
> unlikely to also be a global.)
> 
> But hey! It's fun! :)
> 
> ChrisA
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Feature suggestion: "Using declarations" i.e. context managers ("with" blocks) tied to scope/lifetime of the variable rather than to nesting

[Thomas Jollans]

On 21/02/2019 19:35, mnl.p...@gmail.com wrote:
> (I sent this a few days ago but got bounced without a reason—don’t see it
> posted, so I’m trying one more time.)

No, it got through. And it's in the archive:

https://mail.python.org/pipermail/python-list/2019-February/739548.html

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: revisiting the "What am I running on?" question

[Thomas Jollans]

On 20/02/2019 08.13, eryk sun wrote:
> On 2/19/19, Chris Angelico  wrote:
>>
>> The value "win32" means Windows (or, more technically, "Windows NT
>> family", as opposed to Win95/Win98 - but since Win XP, that's the only
>> type of Windows there is). If you actually care about whether it's a
>> 32-bit or 64-bit OS, you can look at sys.maxsize.
> 
> I didn't start using Python until long after I stopped using Windows
> 9x. Do you recall what sys.platform was for Windows 9x? It was also
> the Win32 API, but layered over a 32-bit VMM that extended/replaced
> 16-bit DOS. There were many differences between it and the NT
> implementation of Win32, such as no security or Unicode, and a
> completely different console subsystem.
> 
> IMO, "win32" was never the right platform name, but we're stuck with
> it. The correct platform name is simply "windows". Win32 is also the
> legacy API name. Nowadays the preferred name is the Windows API, or
> WINAPI for short. Win32 is also the legacy 32-bit ABI name. It's an
> okay name, but I'd prefer if we could change it to "win-ia32", which
> would parallel the 64-bit ABI name "win-amd64".

I'm fairly sure "win32" was used on W9x as well. In any case it *was*
correct at the time, as early versions of Python also ran on DOS and
Windows 3.1. "windows" would not have been suitable.

But yeah, we're stuck with it. There are obviously good reasons that it
wasn't changed when we moved to amd64, but it is annoying.


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Feature suggestions: "Using declarations" i.e. context managers ("with" blocks) tied to scope/lifetime of the variable rather than to nesting

[Thomas Jollans]

On 19/02/2019 05.15, mnl.p...@gmail.com wrote:
> This becomes more ugly if multiple withs get nested.
> 

This is what contextlib.ExitStack is for.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's up with Activestate Python?

[Thomas Jollans]

On 18/02/2019 10.21, Barry Scott wrote:
> 
> 
>> On 18 Feb 2019, at 08:49, Thomas Jollans  wrote:
>>
>> Anaconda also has its moments, and has some packages that PyPI doesn't
>> (for my use case, this is primarily PyQt5).
> 
> Odd I use PyQt5 from PyPI all the time and have for a few years now.

It would appear that my information is outdated!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's up with Activestate Python?

[Thomas Jollans]

On 18/02/2019 10.21, Barry Scott wrote:
> 
> 
>> On 18 Feb 2019, at 08:49, Thomas Jollans  wrote:
>>
>> Anaconda also has its moments, and has some packages that PyPI doesn't
>> (for my use case, this is primarily PyQt5).
> 
> Odd I use PyQt5 from PyPI all the time and have for a few years now.

It would appear that my information is outdated!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What's up with Activestate Python?

[Thomas Jollans]

On 14/02/2019 19:10, Grant Edwards wrote:
> On 2019-02-14, Liste guru  wrote:
>> Il 14/02/2019 00:06, Grant Edwards ha scritto:
>>> For many, many years I've always installed ActiveState's ActivePython
>>> Community edition when forced to use Windows.  It has always included
>>> ...
>>> I guess it's time to switch to Anaconda or ???
>>
>> I've also used the ActiveState python, expecially for the 2.7.x
>> series, mainly for the oflline docs and the pywin32 libraries.
>>
>> Now the situation is better and with pip is really easy to have an 
>> updated python with a lot of libs so there is less need for the 
>> ActiveState distribution.
> 
> How does that work for libraries that require a C compiler?  Does pip
> know how to download and install any required C/C++ toolchains?
> 

Most extension modules on PyPI are distributed with binary wheels these
days, so there's no need for a C compiler. This is certainly the case
for the usual suspects (numpy etc). You probably won't have any trouble
using the python.org version. In a pinch, the Microsoft compiler isn't
hard to install (and free to download).

Anaconda also has its moments, and has some packages that PyPI doesn't
(for my use case, this is primarily PyQt5).

I don't think I've used ActiveState since the Python 2.3 days, so I
can't speak for that.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: get the terminal's size

[Thomas Jollans]

On 14/01/2019 12.57, Alex Ternaute wrote:
> Hi there,
> 
> I want to know the number of columns of the terminal where python2 writes 
> it's outputs.
> 
> In a terminal, I type
> $ echo $COLUMNS
> 100
> 
> But in Python, os.getenv("COLUMNS") gets nothing.
> It gets nothing as well if I try to read the output of "echo $COLUMNS" 
> from a subprocess.
> 
> I feel that I'm missing something but what ?
> 
> Looking on the internet for a hint, I see that python3 has an  
> os.get_terminal_size(). 

Use that then.

> Please, is there something similar for python2 ?

I suspect there is some solution in the curses module...

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Kivy native GUI examples

[Thomas Jollans]

On 07/01/2019 15.51, Dave wrote:
> I need to select a Python GUI.  It needs to cover all of the desktops
> (Linux, Windows, Apple) and hopefully mobile (Android and Ios).  I'm
> looking at Kivy, but have yet to find an example app. that has a native
> looking GUI (Windows, Mac, Linux/Gnome/KDE).  Is that possible and
> anyone know of some examples?

AFAIK looking like a native app is quite simply not something Kivy helps
you with. If that's important, you should look into other options such
as Qt (PyQt5 or PySide2).


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Kivy native GUI examples

[Thomas Jollans]

On 07/01/2019 15.51, Dave wrote:
> I need to select a Python GUI.  It needs to cover all of the desktops
> (Linux, Windows, Apple) and hopefully mobile (Android and Ios).  I'm
> looking at Kivy, but have yet to find an example app. that has a native
> looking GUI (Windows, Mac, Linux/Gnome/KDE).  Is that possible and
> anyone know of some examples?

AFAIK looking like a native app is quite simply not something Kivy helps
you with. If that's important, you should look into other options such
as Qt (PyQt5 or PySide2).


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Creating type evaluation annotation

[Thomas Jollans]

On 06/12/2018 11:48, Marek Mosiewicz wrote:
> I'm Java developer,but had some experience with Python based
> ERP software. It is quite serious application and I feel 
> unconfortable with not having type checking. I do not say language
> should be static, but having checking method signature
> is big win. For example in refactoring.

I get the feeling that you're essentially describing mypy.

> I know that Python 3 has possibility to have indicate
> type for varibale or param
> What could be great to have possibility to annotate any 
> class with what I call "type evaluator" function.
> That could check method signature with rules specific 
> for given type.
> The simplest evaluator would simply check if signature
> is same as method definition. 
> But it could go futher. It could check whatever is valid 
> logic for creating methods in given class and whatever 
> are valid params for given method.
> Even more powerful evaluator could be not only
> valid/not valid indicator. It could validate
> and suggest next param or method part name for given
> written part of call. That could make python IDEs
> much more powerful
> From my experience having signature checking and
> hinting params when writing code is really big win.
> I could simply refactor 10 lines of code in
> Java, but same for Python is much more difficult
> 
> Best regards,
> Marek Mosiewicz
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Import module from file path

[Thomas Jollans]

On 05/12/2018 02:30, Oscar Benjamin wrote:
> Hi all,
> 
> I'm looking to import a module given a string representing the path to
> the .py file defining the module. For example given this setup
> 
> mkdir -p a/b/c
> touch a/__init__.py
> touch a/b/__init__.py
> touch a/b/c/__init__.py
> touch a/b/c/stuff.py
> 
> I have a module a.b.c.stuff which is defined in the file
> '/home/oscar/work/project/a/b/c/stuff.py'. Given that a.b.c.stuff is
> importable and I have the (relative or absolute) path of stuff.py as a
> string I would like to import that module.
> 
> I want this to work in 2.7 and 3.4+ and have come up with the
> following which works for valid inputs:

I might try something along the lines of: (untested)

if not filename.endswith('.py'):
raise ValueError('not a .py file')
abs_filename = os.path.abspath(filename)
for path_root in sys.path:
abs_root = os.path.abspath(path_root)
if abs_filename.startswith(abs_root):
rel_filename = os.path.relpath(abs_filename, abs_root)
if '.' in rel_filename[:-3]:
# '.' in directory names? can't be right!
continue
mod_name = rel_filename[:-3].replace(os.sep, '.')
try:
return importlib.import_module(mod_name)
except ImportError:
continue
else:
raise ValueError('not an importable module')

This should work with namespace packages.


> Also it seems as if there should be a simpler way to get from
> the path to the module name...

I doubt it. There's no reason for the import machinery to have such a
mechanism. Besides, not all modules have real file names (e.g. if they
live in zip files)

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to run an infinite loop on Menu?

[Thomas Jollans]

On 03/12/2018 08:58, huey.y.ji...@gmail.com wrote:
> Hi Folks,
> 
> I need to run an infinite loop on a Menu button, like this:
> 
> from Tkinter import *
> 
> def run_job():
> i = 0
> while 1:
> i = i + 1
> if i > 100:
> break
>  ...
> 
> root = Tk()
> menu = Menu(root)
> root.config(menu=menu)
> filemenu = Menu(menu)
> menu.add_cascade(label="Jobs", menu=filemenu)
> filemenu.add_command(label="Run", command=run_job)
> mainloop()
> 
> 
> In fact, the def run_job has more things to do. But, it is an infinite loop 
> in essence. It works. However, when the def run_job is running, the menu will 
> be hang there. I have no way to stop it. Even if I raise it with 
> Key-Interupt, the thing was the same: the infinite loop cannot be stopped 
> nicely. The only way was to wait until the Python interpreter notice the 
> problem, and popped out an error message, then I can kill the running program.

Basic rule of thumb for GUI programming: never block the main (GUI)
thread, i.e. never run any long-running task in the main thread.

If you want to start a long-running task from a GUI event, start a
worker thread; from there, you could then perhaps update a progress bar
in your loop. If you want to be able to stop the task from the GUI,
you'll want something like a "stop the task ASAP please" flag/variable
set by the stop button that the loop periodically checks.

> 
> Can anybody tell me how to handle this? I wish when variable i reached 100, 
> the run_job will be end, and the Menu "Run" will be free.
> 
> Thanks in advance. 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Injecting methods into instance / class

[Thomas Jollans]

On 02/12/2018 18:56, duncan smith wrote:
> Hello,
>   I have a lot of functions that take an instance of a particular
> class as the first argument. I want to create corresponding methods in
> the class. I have tried the following, which (when called from __init__)
> creates the relevant methods in an instance (Python 3.6).

As it sounds like you're creating the classes you want these methods in
yourself, how about using good old-fashioned inheritance instead?


> def init_methods(self):
> for func_name, method_name in [('add', '__add__'),
>('subtract', '__sub__')]:
> setattr(self, method_name,
> types.MethodType(globals()[func_name], self))
> 
> 
> The problem is that e.g.
> 
> x.__sub__(y)
> 
> works as expected, but
> 
> x - y
> 
> does not (because special methods are looked up in the class rather than
> the instance).
> 
> I have tried to find examples of injecting methods into classes without
> success. I have tried a few things that intuition suggested might work,
> but didn't - like removing the first line above, dedenting and replacing
> self with the class. This is only to save typing and make the code
> cleaner, but I would like to get it right. Any pointers appreciated. TIA.
> 
> Duncan
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: multiple JSON documents in one file, change proposal

[Thomas Jollans]

On 30/11/2018 23:40, Marko Rauhamaa wrote:
> Paul Rubin :
>> Maybe someone can convince me I'm misusing JSON but I often want to
>> write out a file containing multiple records, and it's convenient to
>> use JSON to represent the record data.
>>
>> The obvious way to read a JSON doc from a file is with "json.load(f)"
>> where f is a file handle. Unfortunately, this throws an exception
> 
> I have this "multi-JSON" need quite often. In particular, I exchange
> JSON-encoded messages over byte stream connections. There are many ways
> of doing it. Having rejected different options ( https://en.wikipedia.org/wiki/JSON_streaming>), I settled with
> terminating each JSON value with an ASCII NUL character, which is
> illegal in JSON proper.

FWIW, YAML supports multiple documents in one stream (separated by ---)

If you just want multiple JSON objects in a file (without any streaming
needs), you can just use a JSON array...
> 
>> I also recommend the following article to those not aware of how badly
>> designed JSON is: http://seriot.ch/parsing_json.php
> 
> JSON is not ideal, but compared with XML, it's a godsend.
> 
> What would be ideal? I think S-expressions would come close, but people
> can mess up even them: https://www.ietf.org/rfc/rfc2693.txt>.
> 
> 
> Marko
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python2.7 unicode conundrum

[Thomas Jollans]

On 25/11/2018 18:51, Robert Latest via Python-list wrote:
> Hi folks,
> what semmingly started out as a weird database character encoding mix-up
> could be boiled down to a few lines of pure Python. The source-code
> below is real utf8 (as evidenced by the UTF code point 'c3 a4' in the
> third line of the hexdump). When just printed, the string "s" is
> displayed correctly as 'ä' (a umlaut), but the string representation
> shows that it seems to have been converted to latin-1 'e4' somewhere on
> the way.

It's not being converted to latin-1. It's a unicode string, as evidences
by the 'u'.

u'\xe4' is a unicode string with one character, U+00E4 (ä)

> How can this be avoided?
> 
> dh@jenna:~/python$ cat unicode.py
> # -*- encoding: utf8 -*-
> 
> s = u'ä'
> 
> print(s)
> print((s, ))
> 
> dh@jenna:~/python$ hd unicode.py 
>   23 20 2d 2a 2d 20 65 6e  63 6f 64 69 6e 67 3a 20  |# -*- encoding: |
> 0010  75 74 66 38 20 2d 2a 2d  0a 0a 73 20 3d 20 75 27  |utf8 -*-..s = u'|
> 0020  c3 a4 27 0a 0a 70 72 69  6e 74 28 73 29 0a 70 72  |..'..print(s).pr|
> 0030  69 6e 74 28 28 73 2c 20  29 29 0a 0a  |int((s,))..|
> 003c
> dh@jenna:~/python$ python unicode.py
> ä
> (u'\xe4',)
> dh@jenna:~/python$
> 
> 
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Odd truth result with in and ==

[Thomas Jollans]

On 2018-11-22 09:58, Chris Angelico wrote:
> On Thu, Nov 22, 2018 at 7:51 PM Thomas Jollans  wrote:
>>
>> On 21/11/2018 20:18, Python wrote:
>>> $ python3
>>> Python 3.5.2 (default, Nov 23 2017, 16:37:01)
>>> [GCC 5.4.0 20160609] on linux
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> 1 in [1,2,3] == True
>>> False
>>>>>> 1 in ([1,2,3] == True)
>>> Traceback (most recent call last):
>>>   File "", line 1, in 
>>> TypeError: argument of type 'bool' is not iterable
>>>>>> (1 in [1,2,3]) == True
>>> True
>>>
>>
>> See: https://github.com/cosmologicon/pywat ;-)
>>
> 
> I find it fascinating that quite a few of the Wats given on the
> landing page are specifically poking fun at IEEE floating point
> (completely documented and intended behaviour that exists across many
> languages), yet the "Wat Quiz", also in that repository, specifically
> excludes floats. TRWTF is inconsistently poking fun at a language's
> consistencies.

Clearly the author was struggling to find "wat"s of the sort you get in
Ruby or JavaScript.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about implementing immutability

[Thomas Jollans]

On 2018-11-21 17:45, Iwo Herka wrote:
> Hello,
> 
> Let's say I want to implement immutability for user-defined class.
> More precisely, a class that can be modified only in its (or its
> super-class') __init__ method. My initial idea was to do it the
> following fashion:
> 
> def __setattr__(self, *args, **kwargs):
> if sys._getframe(1).f_code.co_name == '__init__':
> return super().__setattr__(*args, **kwargs)

I don't like this. First of all, it's needlessly cryptic, and secondly
this allows other classes' __init__s to set attributes.

I might try setting a self._fixed flag at the end of init and do a check

if getattr(self, '_fixed', False):
raise TypeError(f"'{type(self)}' is immutable")

but this presumably comes with other problems.

> raise AttributeError()

This should be TypeError.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about implementing immutability

[Thomas Jollans]

On 2018-11-21 21:36, Calvin Spealman wrote:
> If you want to create your own immutable class, maybe inherit from a
> namedtuple?

If you're tempted to go down that route and can require Python 3.7, use
dataclasses!


> 
> On Wed, Nov 21, 2018 at 11:45 AM Iwo Herka  wrote:
> 
>> Hello,
>>
>> Let's say I want to implement immutability for user-defined class.
>> More precisely, a class that can be modified only in its (or its
>> super-class') __init__ method. My initial idea was to do it the
>> following fashion:
>>
>> def __setattr__(self, *args, **kwargs):
>> if sys._getframe(1).f_code.co_name == '__init__':
>> return super().__setattr__(*args, **kwargs)
>> raise AttributeError()
>>
>> What do you think of this approach? Is there a better one?
>> Thanks.
>>
>> Sincerely,
>> Iwo Herka
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Odd truth result with in and ==

[Thomas Jollans]

On 21/11/2018 20:18, Python wrote:
> $ python3
> Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
> [GCC 5.4.0 20160609] on linux
> Type "help", "copyright", "credits" or "license" for more information.
 1 in [1,2,3] == True
> False
 1 in ([1,2,3] == True)
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: argument of type 'bool' is not iterable
 (1 in [1,2,3]) == True
> True
> 

See: https://github.com/cosmologicon/pywat ;-)




signature.asc
Description: OpenPGP digital signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Reading 'scientific' csv using Pandas?

[Thomas Jollans]

On 2018-11-18 19:22, Martin Schöön wrote:
> Den 2018-11-18 skrev Shakti Kumar :
>> On Sun, 18 Nov 2018 at 18:18, Martin Schöön  wrote:
>>>
>>> Now I hit a bump in the road when some of the data is not in plain
>>> decimal notation (xxx,xx) but in 'scientific' (xx,xxxe-xx) notation.
>>>
>>
>> Martin, I believe this should be done by pandas itself while reading
>> the csv file,
>> I took an example in scientific notation and checked this out,
>>
>> my sample.csv file is,
>> col1,col2
>> 1.1,0
>> 10.24e-05,1
>> 9.492e-10,2
>>
> That was a quick answer!
> 
> My pandas is up to date.
> 
> In your example you use the US convention of using "." for decimals
> and "," to separate data. This works perfect for me too.
> 
> However, my data files use European conventions: decimal "," and TAB
> to separate data:
> 
> col1  col2
> 1,1   0
> 10,24e-05 1
> 9,492e-10 2
> 
> I use 
> 
> EUData = pd.read_csv('file.csv', skiprows=1, sep='\t',
> decimal=',', engine='python')
> 
> to read from such files. This works so so. 'Common floats' (3,1415 etc)
> works just fine but 'scientific' stuff (1,6023e23) does not work.
> 
> /Martin
> 


This looks like a bug in the 'python' engine specifically. I suggest you
write a bug report at https://github.com/pandas-dev/pandas/issues

(conda:nb) /tmp
0:jollans@mn70% cat test.csv
Index   Value
0   1,674
1   3,48e+3
2   8,1834e-10
3   3984,109
4   2830812370

(conda:nb) /tmp
0:jollans@mn70% ipython
Python 3.7.0 (default, Oct  9 2018, 10:31:47)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd



In [2]: pd.read_csv('test.csv', header=[0], index_col=0, decimal=',',
sep='\t')

Out[2]:
  Value
Index
0  1.674000e+00
1  3.48e+03
2  8.183400e-10
3  3.984109e+03
4  2.830812e+09

In [3]: pd.read_csv('test.csv', header=[0], index_col=0, decimal=',',
sep='\t', engine='python')

Out[3]:
Value
Index
0   1.674
1 3,48e+3
2  8,1834e-10
33984.109
4  2830812370

In [4]: pd.__version__


Out[4]: '0.23.4'



-- 
Cheers,
 Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: IDLE Default Working Directory

[Thomas Jollans]

On 13/11/2018 00:45, Terry Reedy wrote:
> 
> On Windows, a simple alternate is a .bat file.  I belive the folloiwing
> should work.
> 
> cd c:/desired/startup/directory
> py -x.y -m idlelib
> 
> The default for x.y is latest 3.x or latest 2.x if no 3.x.

Correct me if I'm wrong, but won't that create an empty command prompt
window nobody wants to see? (which you can get rid of by calling
START /B on a win32 application like pythonw.exe. Is there a pyw.exe or
something?)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: distribute python interpreter and dependencies

[Thomas Jollans]

On 12/11/2018 17:40, Juan Cristóbal Quesada wrote:
> Hello,
> this is my first mail. I resorted to the list after some prior struggling.

Welcome!

> Im facing the need to distribute a python installation folder and
> interpreter in a shared network drive.
> 
> Im also distributing the application's source code that would lie also in
> another network drive.
> 
> For this, my attempts have gone towards replicating the python installation
> folder created after installing python27 in one machine and copy all the
> files and directories to the network drive. After that, copied the
> python27.dll of the C:/Windows/System32 file and set all the
> Python27/lib;Python27/DLLs/Python27/Scripts... to the PATH environment
> variable through a launcher script.

I assume you have a good reason to want to use an old version of Python...

> 
> This works on my machine and a couple othersBUT, not in some other
> machines running as well windows 10 pro. 

In what way does it not work? Is there an error message?

> So i investigated a bit and
> discovered that if i install the python27 (2.7.11 same version) in one of
> those failing machines... the "ctypes.pyd" module's filesize is
> different So i replaced the original python27 folders with those of the
> new installed python and now it works on those machines..havent
> tried yet if it still works on the first ones...
> 
> Why is this behaviour? Im guessing the python27 installer generates some
> dlls "on the fly" that are tied to the windows operating system...
> 
> I dont want to create a windows executable via py2exe or
> pyinstaller.. What are the best steps to make a python interpreter
> available to all windows based different machines? Am i missing something
> else? What are the steps the python windows installer performs in order?

I have no idea what the Python.org installer is doing here, but you
could try one of the other Python distributions (e.g. miniconda)...
MAYBE you'll have more luck with that (Or ActivePython, or WinPython, or
whatever).


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Issue in parsing the strings in python code

[Thomas Jollans]

On 2018-11-12 10:23, srinivasan wrote:
> Hi Thomas,
> 
> Great to hear from you, Could you please let me know how do I get the UUID
> "1da7d068-4548-4446-bf88-a440e49db1b1" by passing the name of the SSID
> "Funkloch' using "nmcli --terse" ??


Have a look at the output. It appears to me that the fields are
separated by colons, so you should be able to split each line on ':'.

If you're stuck, let us see what you've tried!

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Issue in parsing the strings in python code

[Thomas Jollans]

On 12/11/2018 09:28, srinivasan wrote:
> Dear Python Experts team,
> 
> This question might be very simple for you, As am newbie to python, could
> you please how to parse the below strings
> 
> [snip]
> 
> 
> root:~/qa/robot_tests# nmcli c show

Pro tip: many *nix tools have a flag that makes them produce
machine-readable output. E.g.: from the nmcli man page:

OPTIONS
   -t | --terse
   Output is terse. This mode is designed and suitable for
computer (script) processing.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Good editor for python

[Thomas Jollans]

On 11/11/2018 10:14, Olive wrote:
> I am not a professional programmer but I use Python regularly for custom 
> scripts (and plot with matplotlib). I have just learned VBA for Excel: what I 
> found amazing was their editor: it is able to suggest on the spot all the 
> methods an object support and there is a well-integrated debugger. I wonder 
> if something similar exists for Python. For now I just use emacs with the 
> command line pdb. What do people use here? Ideally I would like to have 
> something that is cross platform Windows/Linux.

There are several popular full-featured IDEs, like PyCharm and Spyder.
AFAIK both also have some special support for a scientific workflow with
matplotlib integration.

If you're mainly using Python for scripts you invoke interactively to
create plots and such you might enjoy Jupyter Notebook or JupyterLab.
Completion works great, debugging with ipdb is, well, adequate.

For actual "editors", I'm sure there are ways to make vim or emacs
everything you could possibly want, but there are two popular, modern,
cross-platform programmers' editors that, to my mind, stand out at the
moment:

Sublime Text is a fantastic, powerful and (fairly) fast editor
(proprietary, shareware, 80 USD) with good support for Python with a
number of packages (plugins), chief among them "Anaconda", which gives
you linting and autocompletion.

Visual Studio Code (open source, from Microsoft) is younger and a bit
slower and more bloated, but it has fantastic Python support, including
a well-integrated visual debugger.

There are of course loads and loads of other options. Everything I've
mentioned runs at least on x86_64 Linux, OSX, and Windows.


-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: installing modules problem

[Thomas Jollans]

On 2018-11-08 06:04, Ian K. wrote:
> Hello,
> 
> My name is Ian Kilty and I have been having trouble with pip. I have pip
> installed, and I have tried to install modules they say they are installed
> in cmd, but when I go into python and import the module, it can't find it.
> I hope there is a simple solution to this problem, please let me know as
> soon as possible.
> 
> -- from Not A User because I don't want to be made fun of in Tron --
> 

Hi!

some questions:

 - is there any chance you have multiple versions of python installed?
   Could you be using the wrong pip?
 - what operating system are you using?
 - what are you trying to install and use?
 - how exactly are you trying to install and import this module? (exact
   commands, exact copy-pasted output)

-- Thomas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Overwhelmed by the Simplicity of Python. Any Recommendation?

[Thomas Jollans]

On 07/11/2018 21:31, MRAB wrote:
> On 2018-11-07 09:20, Thomas Jollans wrote:
>> On 06/11/2018 22:51, Lie Ryan wrote:
>>>> I like to step through my code line by line,
>>>> it's impossible to do it with
>>>> object-oriented programming language.
>>>
>>> I suggest pudb, it's a curses based debugger, which is nicer than
>>> pdb, but doesn't require tedious IDE setup.
>>
>> I'll just take this opportunity to point out (for those that don't know)
>> that Visual Studio Code (an open source cross-platform programmer's text
>> editor of the same calibre as Sublime or Atom, not an IDE) has great
>> support for (graphical) debugging of Python code. Not tedious to set up,
>> particularly. Obviously there are plugins for other editors, but they're
>> usually not this well-integrated.
>>
> I find that the code does run more slowly, though.

Can't argue with that.

I'm not totally sold on vscode, either. But I enjoy the debugger!


> 
>>>
>>>> Also, there's no good REPL IDE. 
>>>
>>> Not quite sure what you meant by REPL IDE, but did you try IPython
>>>
>>
>> If you find yourself wanting a debugger in an IPython/Jupyter notebook,
>> ipdb is pretty nice.
>>

-- 
https://mail.python.org/mailman/listinfo/python-list