[issue45501] [idea] Successfully creating a venv could print a message.

[Steven D'Aprano]

Steven D'Aprano  added the comment:

This may be an example of the problem:

https://discuss.python.org/t/why-does-venv-not-work-on-my-windows-system/11167

The poster there hasn't responded yet, so it isn't clear what is the actual 
issue is. But it looks suspiciously like a failure to realise that the venv 
actually did work.

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45501] [idea] Successfully creating a venv could print a message.

[Julien Palard]

New submission from Julien Palard :

I realized that many students get surprised by `python -m venv .venv` not 
printing anything, a few even think it completly failed.

I'm OK teaching them this is normal, as `mv`, `cp`, `rm`, `django-admin 
startproject`, ... does not print neither when they succeed.

But I fear many other Python users could be surprised too and not have a 
teacher around to reassure them.

I kept this as a status-quo for years but thinking of it today I though « And 
why not telling them how to activate the venv? »

Would'nt it be great to have:

$ python3 -m venv .venv
Environment created successfully, activate it using:

source .venv/bin/activate

or

PS C:\Users\Admin> python3 -m venv .venv
Environment created successfully, activate it using:

.venv\Scripts\Activate.ps1

and so on?

A `-q`/`--quiet` could be added for scripts creating venvs.

--
messages: 404132
nosy: mdk
priority: normal
severity: normal
status: open
title: [idea] Successfully creating a venv could print a message.

___
Python tracker 
<https://bugs.python.org/issue45501>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: PEP Idea: Real private attribute

[Calvin Spealman]

On Tue, Aug 31, 2021 at 1:19 PM Mehrzad Saremi 
wrote:

> Calvin, even if the language offered truly private members?
>

I'm saying I don't think they're necessary, especially not for the use case
posited here. Private members in other languages are about things internal
to the class of the object itself. If we *did* have them in Python, you
couldn't use private members of an object from a decorator on it because
the decorator is external to the target class.


> On Tue, 31 Aug 2021 at 17:31, Calvin Spealman  wrote:
>
>> The right way for those decorators to hold some private information,
>> imho, isn't to put anything on the decorated object at all, but to use a
>> weak-ref dictionary using the target object as a key.
>>
>> On Sat, Aug 28, 2021 at 5:42 PM Mehrzad Saremi 
>> wrote:
>>
>>> Python currently uses name mangling for double-underscore attributes.
>>> Name
>>> mangling is not an ideal method to avoid name conflicting. There are
>>> various normal programming patterns that can simply cause name
>>> conflicting
>>> in double-underscore members. A typical example is when a class is
>>> re-decorated using the same decorator. The decorator can not take
>>> double-underscore members without name conflicts. For example:
>>>
>>> ```
>>> @custom_decorator("a")
>>> @custom_decorator("b")
>>> class C:
>>> pass
>>> ```
>>>
>>> The `@custom_decorator` wrapper may need to hold private members, but
>>> Python's current name conflict resolution does not provide any solution
>>> and
>>> the decorator cannot hold private members without applying tricky
>>> programming methods.
>>>
>>> Another example is when a class inherits from a base class of the same
>>> name.
>>>
>>> ```
>>> class View:
>>> """A class representing a view of an object; similar to
>>> numpy.ndarray.view"""
>>> pass
>>>
>>> class Object:
>>> class View(View):
>>> """A view class costumized for objects of type Object"""
>>> pass
>>> ```
>>>
>>> Again, in this example, class `Object.View` can't take double-underscore
>>> names without conflicting with `View`'s.
>>>
>>> My idea is to introduce real private members (by which I do not mean to
>>> be
>>> inaccessible from outside the class, but to be guaranteed not to conflict
>>> with other private members of the same object). These private members are
>>> started with triple underscores and are stored in a separate dictionary
>>> named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
>>> will be a double layer dictionary that takes 'type' keys in the first
>>> level, and 'str' keys in the second level.
>>>
>>> For example, assume that the user runs the following code:
>>> ```
>>> class C:
>>> def __init__(self, value):
>>> self.___member = value
>>>
>>> c = C("my value")
>>> ```
>>>
>>> On the last line, Python's attribute setter creates a new entry in the
>>> dictionary with key `C`, adds the value "my value" to a new entry with
>>> the
>>> key 'member'.
>>>
>>> The user can then retrieve `c.___member` by invoking the `__privs__`
>>> dictionary:
>>>
>>> ```
>>> print(c.__privs__[C]['member'])  # prints 'my value'
>>> ```
>>>
>>> Note that, unlike class names, class objects are unique and there will
>>> not
>>> be any conflicts. Python classes are hashable and can be dictionary keys.
>>> Personally, I do not see any disadvantage of using __privs__ over name
>>> mangling/double-underscores. While name mangling does not truly guarantee
>>> conflict resolution, __privs__ does.
>>>
>>> Please discuss the idea, let me know what you think about it, whether
>>> there
>>> are possible disadvantages, and if you think it will be approved as a
>>> PEP.
>>>
>>> Thanks,
>>> Mehrzad Saremi
>>>
>>> AI M.Sc. grad. from AUT
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>>>
>>>
>>
>> --
>>
>> CALVIN SPEALMAN
>>
>> SENIOR QUALITY ENGINEER
>>
>> calvin.speal...@redhat.com  M: +1.336.210.5107
>> [image: https://red.ht/sig] <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>
>

-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

calvin.speal...@redhat.com  M: +1.336.210.5107
[image: https://red.ht/sig] <https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Mehrzad Saremi]

Calvin, even if the language offered truly private members?

On Tue, 31 Aug 2021 at 17:31, Calvin Spealman  wrote:

> The right way for those decorators to hold some private information, imho,
> isn't to put anything on the decorated object at all, but to use a weak-ref
> dictionary using the target object as a key.
>
> On Sat, Aug 28, 2021 at 5:42 PM Mehrzad Saremi 
> wrote:
>
>> Python currently uses name mangling for double-underscore attributes. Name
>> mangling is not an ideal method to avoid name conflicting. There are
>> various normal programming patterns that can simply cause name conflicting
>> in double-underscore members. A typical example is when a class is
>> re-decorated using the same decorator. The decorator can not take
>> double-underscore members without name conflicts. For example:
>>
>> ```
>> @custom_decorator("a")
>> @custom_decorator("b")
>> class C:
>> pass
>> ```
>>
>> The `@custom_decorator` wrapper may need to hold private members, but
>> Python's current name conflict resolution does not provide any solution
>> and
>> the decorator cannot hold private members without applying tricky
>> programming methods.
>>
>> Another example is when a class inherits from a base class of the same
>> name.
>>
>> ```
>> class View:
>> """A class representing a view of an object; similar to
>> numpy.ndarray.view"""
>> pass
>>
>> class Object:
>> class View(View):
>> """A view class costumized for objects of type Object"""
>> pass
>> ```
>>
>> Again, in this example, class `Object.View` can't take double-underscore
>> names without conflicting with `View`'s.
>>
>> My idea is to introduce real private members (by which I do not mean to be
>> inaccessible from outside the class, but to be guaranteed not to conflict
>> with other private members of the same object). These private members are
>> started with triple underscores and are stored in a separate dictionary
>> named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
>> will be a double layer dictionary that takes 'type' keys in the first
>> level, and 'str' keys in the second level.
>>
>> For example, assume that the user runs the following code:
>> ```
>> class C:
>> def __init__(self, value):
>> self.___member = value
>>
>> c = C("my value")
>> ```
>>
>> On the last line, Python's attribute setter creates a new entry in the
>> dictionary with key `C`, adds the value "my value" to a new entry with the
>> key 'member'.
>>
>> The user can then retrieve `c.___member` by invoking the `__privs__`
>> dictionary:
>>
>> ```
>> print(c.__privs__[C]['member'])  # prints 'my value'
>> ```
>>
>> Note that, unlike class names, class objects are unique and there will not
>> be any conflicts. Python classes are hashable and can be dictionary keys.
>> Personally, I do not see any disadvantage of using __privs__ over name
>> mangling/double-underscores. While name mangling does not truly guarantee
>> conflict resolution, __privs__ does.
>>
>> Please discuss the idea, let me know what you think about it, whether
>> there
>> are possible disadvantages, and if you think it will be approved as a PEP.
>>
>> Thanks,
>> Mehrzad Saremi
>>
>> AI M.Sc. grad. from AUT
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>>
>
> --
>
> CALVIN SPEALMAN
>
> SENIOR QUALITY ENGINEER
>
> calvin.speal...@redhat.com  M: +1.336.210.5107
> [image: https://red.ht/sig] <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Calvin Spealman]

The right way for those decorators to hold some private information, imho,
isn't to put anything on the decorated object at all, but to use a weak-ref
dictionary using the target object as a key.

On Sat, Aug 28, 2021 at 5:42 PM Mehrzad Saremi 
wrote:

> Python currently uses name mangling for double-underscore attributes. Name
> mangling is not an ideal method to avoid name conflicting. There are
> various normal programming patterns that can simply cause name conflicting
> in double-underscore members. A typical example is when a class is
> re-decorated using the same decorator. The decorator can not take
> double-underscore members without name conflicts. For example:
>
> ```
> @custom_decorator("a")
> @custom_decorator("b")
> class C:
> pass
> ```
>
> The `@custom_decorator` wrapper may need to hold private members, but
> Python's current name conflict resolution does not provide any solution and
> the decorator cannot hold private members without applying tricky
> programming methods.
>
> Another example is when a class inherits from a base class of the same
> name.
>
> ```
> class View:
> """A class representing a view of an object; similar to
> numpy.ndarray.view"""
> pass
>
> class Object:
> class View(View):
> """A view class costumized for objects of type Object"""
> pass
> ```
>
> Again, in this example, class `Object.View` can't take double-underscore
> names without conflicting with `View`'s.
>
> My idea is to introduce real private members (by which I do not mean to be
> inaccessible from outside the class, but to be guaranteed not to conflict
> with other private members of the same object). These private members are
> started with triple underscores and are stored in a separate dictionary
> named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
> will be a double layer dictionary that takes 'type' keys in the first
> level, and 'str' keys in the second level.
>
> For example, assume that the user runs the following code:
> ```
> class C:
> def __init__(self, value):
> self.___member = value
>
> c = C("my value")
> ```
>
> On the last line, Python's attribute setter creates a new entry in the
> dictionary with key `C`, adds the value "my value" to a new entry with the
> key 'member'.
>
> The user can then retrieve `c.___member` by invoking the `__privs__`
> dictionary:
>
> ```
> print(c.__privs__[C]['member'])  # prints 'my value'
> ```
>
> Note that, unlike class names, class objects are unique and there will not
> be any conflicts. Python classes are hashable and can be dictionary keys.
> Personally, I do not see any disadvantage of using __privs__ over name
> mangling/double-underscores. While name mangling does not truly guarantee
> conflict resolution, __privs__ does.
>
> Please discuss the idea, let me know what you think about it, whether there
> are possible disadvantages, and if you think it will be approved as a PEP.
>
> Thanks,
> Mehrzad Saremi
>
> AI M.Sc. grad. from AUT
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>

-- 

CALVIN SPEALMAN

SENIOR QUALITY ENGINEER

calvin.speal...@redhat.com  M: +1.336.210.5107
[image: https://red.ht/sig] <https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Mehrzad Saremi]

The proposed semantics would be the same as self.__privs__[__class__,
"foo"]; yes I can say the problem is ugliness. The following is an example
where name mangling can be problematic (of course there are workarounds,
yet if double-underscores are meant to represent class-specific members,
the following behavior is an infringement of the purpose).

```
class Weighable:
def weight(self):
raise NotImplementedError()


class AddWeight:
def __init__(self, weight):
self.weight = weight

def __call__(self, cls):
class Wrapper(cls, Weighable):
__weight = self.weight

def weight(self):
return self.__weight + (cls.weight(self) if issubclass(cls, Weighable) else
0)  # Unexpected behavior

return Wrapper


@AddWeight(2.0)
@AddWeight(1.0)
class C:
pass


print(C().weight())
```

> If the parent class isn't aware which class you're in, how is the
language going to define it?

I mean if you want to implement self.__privs__[__class__, "foo"] in a
parent class using __setattr__/__getattribute__ the __class__ value is
unknown.


On Mon, 30 Aug 2021 at 00:26, Chris Angelico  wrote:

> On Mon, Aug 30, 2021 at 5:49 AM Mehrzad Saremi 
> wrote:
> >
> > No, a class ("the class that I'm lexically inside") cannot be accessed
> from
> > outside of the class. This is why I'm planning to offer it as a core
> > feature because only the parser would know. There's apparently no elegant
> > solution if you want to implement it yourself. You'll need to write
> > self.__privs__[__class__, "foo"], whenever you want to use the feature
> and
> > even wrapping it in superclasses won't remedy it, because the parent
> class
> > isn't aware which class you're inside. It seems to me name mangling must
> > have been an ad-hoc solution in a language that it doesn't really fit
> when
> > it could have been implemented in a much more cogent way.
> >
>
> If the parent class isn't aware which class you're in, how is the
> language going to define it?
>
> Can you give a full run-down of the semantics of your proposed privs,
> and how it's different from something like you just used above -
> self.__privs__[__class__, "foo"] - ? If the problem is the ugliness
> alone, then say so; but also, how this would work with decorators,
> since you specifically mention them as a use-case.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Chris Angelico]

On Mon, Aug 30, 2021 at 5:49 AM Mehrzad Saremi  wrote:
>
> No, a class ("the class that I'm lexically inside") cannot be accessed from
> outside of the class. This is why I'm planning to offer it as a core
> feature because only the parser would know. There's apparently no elegant
> solution if you want to implement it yourself. You'll need to write
> self.__privs__[__class__, "foo"], whenever you want to use the feature and
> even wrapping it in superclasses won't remedy it, because the parent class
> isn't aware which class you're inside. It seems to me name mangling must
> have been an ad-hoc solution in a language that it doesn't really fit when
> it could have been implemented in a much more cogent way.
>

If the parent class isn't aware which class you're in, how is the
language going to define it?

Can you give a full run-down of the semantics of your proposed privs,
and how it's different from something like you just used above -
self.__privs__[__class__, "foo"] - ? If the problem is the ugliness
alone, then say so; but also, how this would work with decorators,
since you specifically mention them as a use-case.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Mehrzad Saremi]

No, a class ("the class that I'm lexically inside") cannot be accessed from
outside of the class. This is why I'm planning to offer it as a core
feature because only the parser would know. There's apparently no elegant
solution if you want to implement it yourself. You'll need to write
self.__privs__[__class__, "foo"], whenever you want to use the feature and
even wrapping it in superclasses won't remedy it, because the parent class
isn't aware which class you're inside. It seems to me name mangling must
have been an ad-hoc solution in a language that it doesn't really fit when
it could have been implemented in a much more cogent way.

Best,
[image: image.gif][image: image.gif]

Mehrzad


On Sun, 29 Aug 2021 at 02:18, Chris Angelico  wrote:

> On Sun, Aug 29, 2021 at 7:40 AM Mehrzad Saremi 
> wrote:
> >
> > Python currently uses name mangling for double-underscore attributes.
> Name
> > mangling is not an ideal method to avoid name conflicting. There are
> > various normal programming patterns that can simply cause name
> conflicting
> > in double-underscore members. A typical example is when a class is
> > re-decorated using the same decorator. The decorator can not take
> > double-underscore members without name conflicts. For example:
> >
> > ```
> > @custom_decorator("a")
> > @custom_decorator("b")
> > class C:
> > pass
> > ```
> >
> > The `@custom_decorator` wrapper may need to hold private members, but
> > Python's current name conflict resolution does not provide any solution
> and
> > the decorator cannot hold private members without applying tricky
> > programming methods.
> >
> > Another example is when a class inherits from a base class of the same
> name.
> >
> > ```
> > class View:
> > """A class representing a view of an object; similar to
> > numpy.ndarray.view"""
> > pass
> >
> > class Object:
> > class View(View):
> > """A view class costumized for objects of type Object"""
> > pass
> > ```
> >
> > Again, in this example, class `Object.View` can't take double-underscore
> > names without conflicting with `View`'s.
> >
> > My idea is to introduce real private members (by which I do not mean to
> be
> > inaccessible from outside the class, but to be guaranteed not to conflict
> > with other private members of the same object). These private members are
> > started with triple underscores and are stored in a separate dictionary
> > named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
> > will be a double layer dictionary that takes 'type' keys in the first
> > level, and 'str' keys in the second level.
> >
> > For example, assume that the user runs the following code:
> > ```
> > class C:
> > def __init__(self, value):
> > self.___member = value
> >
> > c = C("my value")
> > ```
> >
> > On the last line, Python's attribute setter creates a new entry in the
> > dictionary with key `C`, adds the value "my value" to a new entry with
> the
> > key 'member'.
> >
> > The user can then retrieve `c.___member` by invoking the `__privs__`
> > dictionary:
> >
> > ```
> > print(c.__privs__[C]['member'])  # prints 'my value'
> > ```
> >
> > Note that, unlike class names, class objects are unique and there will
> not
> > be any conflicts. Python classes are hashable and can be dictionary keys.
> > Personally, I do not see any disadvantage of using __privs__ over name
> > mangling/double-underscores. While name mangling does not truly guarantee
> > conflict resolution, __privs__ does.
>
> Not entirely sure how it would know the right type to use (subclassing
> makes that tricky), but whatever your definition is, there's nothing
> stopping you from doing it yourself. Don't forget that you have
> __class__ available if you need to refer to "the class that I'm
> lexically inside" (that's how the zero-arg super() function works), so
> you might do something like self.__privs__[__class__, "foo"] to refer
> to a thing.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Real private attribute

[Chris Angelico]

On Sun, Aug 29, 2021 at 7:40 AM Mehrzad Saremi  wrote:
>
> Python currently uses name mangling for double-underscore attributes. Name
> mangling is not an ideal method to avoid name conflicting. There are
> various normal programming patterns that can simply cause name conflicting
> in double-underscore members. A typical example is when a class is
> re-decorated using the same decorator. The decorator can not take
> double-underscore members without name conflicts. For example:
>
> ```
> @custom_decorator("a")
> @custom_decorator("b")
> class C:
> pass
> ```
>
> The `@custom_decorator` wrapper may need to hold private members, but
> Python's current name conflict resolution does not provide any solution and
> the decorator cannot hold private members without applying tricky
> programming methods.
>
> Another example is when a class inherits from a base class of the same name.
>
> ```
> class View:
> """A class representing a view of an object; similar to
> numpy.ndarray.view"""
> pass
>
> class Object:
> class View(View):
> """A view class costumized for objects of type Object"""
> pass
> ```
>
> Again, in this example, class `Object.View` can't take double-underscore
> names without conflicting with `View`'s.
>
> My idea is to introduce real private members (by which I do not mean to be
> inaccessible from outside the class, but to be guaranteed not to conflict
> with other private members of the same object). These private members are
> started with triple underscores and are stored in a separate dictionary
> named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
> will be a double layer dictionary that takes 'type' keys in the first
> level, and 'str' keys in the second level.
>
> For example, assume that the user runs the following code:
> ```
> class C:
> def __init__(self, value):
> self.___member = value
>
> c = C("my value")
> ```
>
> On the last line, Python's attribute setter creates a new entry in the
> dictionary with key `C`, adds the value "my value" to a new entry with the
> key 'member'.
>
> The user can then retrieve `c.___member` by invoking the `__privs__`
> dictionary:
>
> ```
> print(c.__privs__[C]['member'])  # prints 'my value'
> ```
>
> Note that, unlike class names, class objects are unique and there will not
> be any conflicts. Python classes are hashable and can be dictionary keys.
> Personally, I do not see any disadvantage of using __privs__ over name
> mangling/double-underscores. While name mangling does not truly guarantee
> conflict resolution, __privs__ does.

Not entirely sure how it would know the right type to use (subclassing
makes that tricky), but whatever your definition is, there's nothing
stopping you from doing it yourself. Don't forget that you have
__class__ available if you need to refer to "the class that I'm
lexically inside" (that's how the zero-arg super() function works), so
you might do something like self.__privs__[__class__, "foo"] to refer
to a thing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

PEP Idea: Real private attribute

[Mehrzad Saremi]

Python currently uses name mangling for double-underscore attributes. Name
mangling is not an ideal method to avoid name conflicting. There are
various normal programming patterns that can simply cause name conflicting
in double-underscore members. A typical example is when a class is
re-decorated using the same decorator. The decorator can not take
double-underscore members without name conflicts. For example:

```
@custom_decorator("a")
@custom_decorator("b")
class C:
pass
```

The `@custom_decorator` wrapper may need to hold private members, but
Python's current name conflict resolution does not provide any solution and
the decorator cannot hold private members without applying tricky
programming methods.

Another example is when a class inherits from a base class of the same name.

```
class View:
"""A class representing a view of an object; similar to
numpy.ndarray.view"""
pass

class Object:
class View(View):
"""A view class costumized for objects of type Object"""
pass
```

Again, in this example, class `Object.View` can't take double-underscore
names without conflicting with `View`'s.

My idea is to introduce real private members (by which I do not mean to be
inaccessible from outside the class, but to be guaranteed not to conflict
with other private members of the same object). These private members are
started with triple underscores and are stored in a separate dictionary
named `__privs__`. Unlike `__dict__` that takes 'str' keys, `__privs__`
will be a double layer dictionary that takes 'type' keys in the first
level, and 'str' keys in the second level.

For example, assume that the user runs the following code:
```
class C:
def __init__(self, value):
self.___member = value

c = C("my value")
```

On the last line, Python's attribute setter creates a new entry in the
dictionary with key `C`, adds the value "my value" to a new entry with the
key 'member'.

The user can then retrieve `c.___member` by invoking the `__privs__`
dictionary:

```
print(c.__privs__[C]['member'])  # prints 'my value'
```

Note that, unlike class names, class objects are unique and there will not
be any conflicts. Python classes are hashable and can be dictionary keys.
Personally, I do not see any disadvantage of using __privs__ over name
mangling/double-underscores. While name mangling does not truly guarantee
conflict resolution, __privs__ does.

Please discuss the idea, let me know what you think about it, whether there
are possible disadvantages, and if you think it will be approved as a PEP.

Thanks,
Mehrzad Saremi

AI M.Sc. grad. from AUT
-- 
https://mail.python.org/mailman/listinfo/python-list

[ANN] pyastinterp - Tree-walking interpreter for Python (+ "advent of code" idea)

[Paul Sokolovsky]

Hello,

pyastinterp is a tree-walking (aka meta-circular) interpreter for
Python. It was the most complete tree-walking interpreter as of a year
ago. As it was written as a personal "advent of code" project during
the winter holidays of 2019/2020.

I remembered about it this holiday season, and decided to post about
it, both as a holiday eyecandy, and as a challenge to other Python
hackers (such winter holidays challenges are known as "advent of
code" (of course, unless you believe there's only one of them)): can you
write a more complete implementation? Or maybe you did during this
year? Please post about it somewhere (perhaps, python-ideas mailing
list?).

https://github.com/pfalcon/pyastinterp


README with more details follows:

pyastinterp
===

This is an AST interpreter (also known as tree-walking interpreter) for
the Python programming language, written in Python itself. It is thus
a meta-circular interpreter. While Python implementations of a Python VM
(i.e. bytecode interpreters) are relatively well known (starting with
PyPy), AST interpreters are much less common beyond those which implement
a very limited subset of the syntax.

Pyastinterp strives to fulfil 2 goals:

1. Be as complete as possible (and see how much effort that takes and
   if there're any limits).
2. Yet be as minimal as possible, avoiding useless boilerplate and
   reimplementing what may be already available.

First goal is pretty obvious: ideally, we'd like Pyastinterp to be able
to interpret (run) any application a native Python interpreter can.

Second goal is rooted in many reasons. To start with, Pyastinterp is
a spiritual part of the [Pycopy](https://github.com/pfalcon/pycopy)
project, whose goal is the development of minimalist Python-compatible
ecosystem. Besides, "bloated" meta-circular interpreters are well out
there (the same PyPy) - there's lack on the opposite side of spectrum.
Pyastinterp is also intended to be a "teaching" and "for-research"
project. Surely, it's much easier to learn and reuse/repurpose a small,
rather than large, project. These last requirements however also put
bounds on "minimality": `pyastinterp` tries to have a clear focus
(AST interpretation) and avoid unneeded boilerplate, but not be
obfuscated, and actually tries to be clear and easily hackable (provide
some reusable infrastructure), even at the expense of somewhat less
optimality/performance.

To achieve the goals of minimality, `pyastinterp` does following:

1. Is concerned with syntax, not semantics (largely).
2. That means that it tries to interpret Python AST tree, but
   relies on the underlying Python runtime to implement the actual
   operations (semantics). In some cases `pyastinterp` actually
   has to deal with runtime semantics, but the point is that it
   should be done only when there's no other choice (i.e. rarely).
3. This in particular means that `pyastinterp` itself requires a
   (pretty) full Python implementation to run, it's not the case
   that it's written in "some subset of Python". (Contrast that
   with e.g. PyPy's RPython, which is a Python subset language,
   in which interpreters are written).
4. Another consequence is that there's no semantic separation
   between "metalevel" (level of the interpreter) and target
   application level. This is again unlike PyPy, where 2 are explicitly
   separated. Lack of separation allows to save on "marshalling"
   values between the 2 levels, but also make it possible to have
   "leaks" between levels, e.g. unexpected and unhandled exception
   in the interpreter to be delivered to the application, causing
   havoc and confusion. Pyastinterp's response to this concern
   is a pledge that there shouldn't be such unhandled interpreter-level
   exceptions. Of course, catching all such cases may take quite
   a lot of testing and debugging, and fixing them may affect code
   minimality and clarity. We'll see how it goes.
5. However, due to choices described above, implementation of many
   syntactic constructs is very clear, concise, and isomorphic
   between the target and meta levels. See e.g. how "if" statement
   is implemented: the implementation looks almost exactly how the
   usage of the "if" statement is.


Supported Python subset
---

Pyastinterp should implement almost all Python3.5 features, stopping
at generators, which are not that easy to implement in a tree-walking
interpreter. It also does not include import hook, so the imported
modules are interpreted by the underlying Python VM, not by
Pyastinterp.


Running
---

Quick way to run something interesting:

wget 
https://raw.githubusercontent.com/python/cpython/3.6/Lib/test/pystone.py
python3 -m astinterp pystone.my 1

You can pass any other script on the command line and see how it goes.


Credits and licensing
-

Pyastinterp is (c) [Paul Sokolovsky](https://github.com/pfalcon) and
is released under the MIT license.



-- 
Best regards,
 Paul 

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Marco Sulla]

On Thu, 19 Mar 2020 at 16:46, Peter J. Holzer  wrote:
> This is similar to algebraic expressions: Have you ever tried to read a
> mathematical paper from before the time the current notation (which we
> Long, convoluted
> sentences instead of what can now be written as a short formula.

...yes, and as an ex Physics student I have to say that short formulas
are completely unreadable, until someone explain them to you. Like
poetry.

> Would it be clearer in this case? I think so. Especially if you have
> several keys to extract.

Ok, but what is the use case? I mean, how much time you need more than
2, 3 keys?

In my Python implementation of frozendict, I'm doing a different
approach for have something as you want: implement the `set` API. For
example:

d = {
'first_name': 'Frances',
'last_name': 'Allen',
'email': 'fal...@ibm.com'
}

d & ("first_name", "last_name") == {'first_name': 'Frances',
'last_name': 'Allen'}

https://github.com/Marco-Sulla/python-frozendict/commit/b2628e14f3275c6ba488dde220023c14f6a8435a

> you can't easily extend a
> language which is too simple.

So you can't easily extend Python? ^^
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[MRAB]

On 2020-03-19 15:17, Musbur wrote:

Hello,

either it's me or everybody else who's missing the point. I understand
the OP's proposal like this:

dict[set] == {k: dict[k] for k in set}
list[iterable] == [list[i] for i in iterable]

Am I right?


"Iterable" is too broad because it includes tuples and strings.
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Rhodri James]

On 19/03/2020 14:47, Peter J. Holzer wrote:

On 2020-03-19 14:24:35 +, Rhodri James wrote:

On 19/03/2020 13:00, Peter J. Holzer wrote:

It's more compact, especially, if "d" isn't a one-character variable,
but an expression:

  fname, lname = db[people].employee.object.get(pk=1234)[['first_name', 
'last_name']]

vs.

  fname = db[people].employee.object.get(pk=1234)['first_name']
  lname = db[people].employee.object.get(pk=1234)['last_name']


I have to say I don't think that's more compact at all.  It's too wide
to be compact.


But 83 characters is still more compact than 121 characters.


Only if your sole metric is a strict character count.  Width matters in 
perception; two shorter lines are easier to take in than one long line, 
even if the long line contains fewer characters.


Besides, terseness isn't one of Python's objectives.

--
Rhodri James *-* Kynesim Ltd
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Chris Angelico]

On Fri, Mar 20, 2020 at 2:46 AM Peter J. Holzer  wrote:
> > A good language has a small core and extensibility via
> > libraries.
>
> This would actually be a feature of the (standard) library.

I think the line kinda blurs here. This would be a feature of a core
data type, and in CPython, it would be implemented using a slot
function in dictobject.c. But it doesn't need any syntactic changes.
Are the core data types part of the language or the standard library?

In any case, it doesn't much matter. The backward compatibility
considerations would be the same either way, and this would mostly be
used with literals and immediate unpacking.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[David Raymond]

For dictionaries it'd even be more useful:
d = {
'first_name': 'Frances',
'last_name': 'Allen',
'email': 'fal...@ibm.com'
}
fname, lname = d[['first_name', 'last_name']]

Why not do this?

import operator
first_last = operator.itemgetter("first_name", "last_name")

fname, lname = first_last(d)


https://docs.python.org/3.8/library/operator.html#operator.itemgetter

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Chris Angelico]

On Fri, Mar 20, 2020 at 2:37 AM Terry Reedy  wrote:
>
> On 3/18/2020 10:28 PM, Santiago Basulto wrote:
>
> > For dictionaries it'd even be more useful:
> >  d = {
> >  'first_name': 'Frances',
> >  'last_name': 'Allen',
> >  'email': 'fal...@ibm.com'
> >  }
> >  fname, lname = d[['first_name', 'last_name']]
>
> Insert ordered dicts make this sort of thing less useful.
>
>  >>> d = {
>  'first_name': 'Frances',
>  'last_name': 'Allen',
>  'email': 'fal...@ibm.com'
> }
>  >>> fname, lname, _ = d
>  >>> fname, lname
> ('first_name', 'last_name')
>

Uhh, great, but the OP wanted fname = "Frances" and lname = "Allen" :)

But even using d.values() here isn't really a solution. It's assuming
positional information where the code really wants to be grabbing
named attributes.

JavaScript has an elegant syntax for object destructuring:

let { first_name, last_name } = d

Python can do this for positional unpacking, but not for named
attributes. The OP's proposal would allow the existing unpacking
syntax to be used for this, simply by returning an unpackable entity
(a list or tuple).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Peter J. Holzer]

On 2020-03-19 08:05:18 -0700, Dan Stromberg wrote:
> On Thu, Mar 19, 2020 at 7:47 AM Peter J. Holzer  wrote:
> > On 2020-03-19 14:24:35 +, Rhodri James wrote:
> > > On 19/03/2020 13:00, Peter J. Holzer wrote:
> > > > It's more compact, especially, if "d" isn't a one-character variable,
> > > > but an expression:
> > > >
> > > >  fname, lname =
> > > >db[people].employee.object.get(pk=1234)[['first_name', 
> > > > 'last_name']]
> > > >
> > > > vs.
> > > >
> > > >  fname = db[people].employee.object.get(pk=1234)['first_name']
> > > >  lname = db[people].employee.object.get(pk=1234)['last_name']
> > >
> > > I have to say I don't think that's more compact at all.  It's too wide
> > > to be compact.
> >
> > But 83 characters is still more compact than 121 characters.
> >
> 
> It's smaller (in a way), but is it more clear?

"Compact" is not the same word as "clear". I was claiming that it was
more compact, not that it was more clear. You can argue that it is less
clear, but you (well, Rhodri) can't redefine the meaning of "compact".

> Compare to regular expressions: they're small, but they tend to be a
> mess.

Usually because what they describe is complex. If you had to write a
matcher equivalent to a complex regexp by hand using basic string
operations, it would probably be harder to read. That would be many
lines of Python code and the reader would have to keep a lot of state
in their head while reading it.

So I would say that regular expressions are generally more clear than
the equivalent long-hand code.

This is similar to algebraic expressions: Have you ever tried to read a
mathematical paper from before the time the current notation (which we
of course now all learn in school) was invented? Long, convoluted
sentences instead of what can now be written as a short formula.

So sometimes "more compact" implies "clearer". Not always of course. The
Obfuscated C Code Contest is full of code which is very short but also
very hard to understand.

Would it be clearer in this case? I think so. Especially if you have
several keys to extract. In Python I would probably use a list or dict
comprehension, which probably isn't much longer, but it is an explicit
loop, so the reader has to think about the loop logic, it isn't "just
built in".


> Remember that the complexity of a programming a language varies with the
> square of its number of features.  Complex languages are not good
> languages.

I somewhat disagree with this. Languages which are too simple (think
"Turing machine") make it very hard to write useful programs, and
probably even harder to read them. As languages aquire more useful
features, they get easier to use and easier to read. At some point this
reverses again, especially if features are inconsistent or interact in
surprising ways. However, while you can just not use some features of a
language which is too complex for you, you can't easily extend a
language which is too simple.

> A good language has a small core and extensibility via
> libraries.

This would actually be a feature of the (standard) library.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Ethan Furman]

On 03/19/2020 02:09 AM, Terry Reedy wrote:

On 3/18/2020 10:28 PM, Santiago Basulto wrote:


For dictionaries it'd even be more useful:
 d = {
 'first_name': 'Frances',
 'last_name': 'Allen',
 'email': 'fal...@ibm.com'
 }
 fname, lname = d[['first_name', 'last_name']]


Insert ordered dicts make this sort of thing less useful.

 >>> d = {
     'first_name': 'Frances',
     'last_name': 'Allen',
     'email': 'fal...@ibm.com'
}
 >>> fname, lname, _ = d
 >>> fname, lname
('first_name', 'last_name')


I disagree -- the ordered dict would have to be small (maybe five entries) and 
one would have to know the insertion order.

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Terry Reedy]

On 3/18/2020 10:28 PM, Santiago Basulto wrote:


For dictionaries it'd even be more useful:
 d = {
 'first_name': 'Frances',
 'last_name': 'Allen',
 'email': 'fal...@ibm.com'
 }
 fname, lname = d[['first_name', 'last_name']]


Insert ordered dicts make this sort of thing less useful.

>>> d = {
'first_name': 'Frances',
'last_name': 'Allen',
'email': 'fal...@ibm.com'
}
>>> fname, lname, _ = d
>>> fname, lname
('first_name', 'last_name')



--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Musbur]

Hello,

either it's me or everybody else who's missing the point. I understand 
the OP's proposal like this:


dict[set] == {k: dict[k] for k in set}
list[iterable] == [list[i] for i in iterable]

Am I right?
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Pieter van Oostrum]

Santiago Basulto  writes:

> Hello community. I have an idea to share with the list to see what you all
> think about it.
>
> I happen to use both Python for Data Science (with our regular friends
> NumPy and Pandas) as well as for scripting and backend development. Every
> time I'm working in server-side Python (not the PyData stack), I find
> myself missing A LOT features from NumPy, like fancy indexing or boolean
> arrays.
>
> So, has it ever been considered to bake into Python's builtin list and
> dictionary types functionality inspired by NumPy? I think multi indexing
> alone would be huge addition. A few examples:
>
> For lists and tuples:
> >>> l = ['a', 'b', 'c']
> >>> l[[0, -1]]
> ['a', 'c']
>
> For dictionaries it'd even be more useful:
> d = {
> 'first_name': 'Frances',
> 'last_name': 'Allen',
> 'email': 'fal...@ibm.com'
> }
> fname, lname = d[['first_name', 'last_name']]
>
> I really like the syntax of boolean arrays too, but considering we have
> list comprehensions, seems a little more difficult to sell.
>
> I'd love to see if this is something people would support, and see if
> there's room to submit a PEP.

How about implementing it yourself:

In [35]: class MultiDict(dict):
 ... def __getitem__(self, idx):
 ... if isinstance(idx, list):
 ... return [self[i] for i in idx]
 ... return super().__getitem__(idx)

In [36]: d = MultiDict({
 ... 'first_name': 'Frances',
 ... 'last_name': 'Allen',
 ... 'email': 'fal...@ibm.com'
 ... })

In [37]: d
Out[37]: {'first_name': 'Frances', 'last_name': 'Allen', 'email': 
'fal...@ibm.com'}

In [38]: d['email']
Out[38]: 'fal...@ibm.com'

In [39]: d[['first_name', 'last_name']]
Out[39]: ['Frances', 'Allen']

The implementation of other methods, like __setitem__ and __init__, and maybe 
some more subtle details like exceptions, is left as an exercise for the reader.
-- 
Pieter van Oostrum
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Dan Stromberg]

On Thu, Mar 19, 2020 at 7:47 AM Peter J. Holzer  wrote:

> On 2020-03-19 14:24:35 +, Rhodri James wrote:
> > On 19/03/2020 13:00, Peter J. Holzer wrote:
> > > It's more compact, especially, if "d" isn't a one-character variable,
> > > but an expression:
> > >
> > >  fname, lname =
> db[people].employee.object.get(pk=1234)[['first_name', 'last_name']]
> > >
> > > vs.
> > >
> > >  fname = db[people].employee.object.get(pk=1234)['first_name']
> > >  lname = db[people].employee.object.get(pk=1234)['last_name']
> >
> > I have to say I don't think that's more compact at all.  It's too wide
> > to be compact.
>
> But 83 characters is still more compact than 121 characters.
>

It's smaller (in a way), but is it more clear?  Compare to regular
expressions: they're small, but they tend to be a mess.

Remember that the complexity of a programming a language varies with the
square of its number of features.  Complex languages are not good
languages.  A good language has a small core and extensibility via
libraries.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Peter J. Holzer]

On 2020-03-19 14:24:35 +, Rhodri James wrote:
> On 19/03/2020 13:00, Peter J. Holzer wrote:
> > It's more compact, especially, if "d" isn't a one-character variable,
> > but an expression:
> > 
> >  fname, lname = db[people].employee.object.get(pk=1234)[['first_name', 
> > 'last_name']]
> > 
> > vs.
> > 
> >  fname = db[people].employee.object.get(pk=1234)['first_name']
> >  lname = db[people].employee.object.get(pk=1234)['last_name']
> 
> I have to say I don't think that's more compact at all.  It's too wide
> to be compact.

But 83 characters is still more compact than 121 characters.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Rhodri James]

On 19/03/2020 13:00, Peter J. Holzer wrote:

On 2020-03-19 18:22:34 +1300, DL Neil via Python-list wrote:

On 19/03/20 3:28 PM, Santiago Basulto wrote:

myself missing A LOT features from NumPy, like fancy indexing or
boolean arrays.
So, has it ever been considered to bake into Python's builtin list and
dictionary types functionality inspired by NumPy? I think multi indexing
alone would be huge addition. A few examples:

[snip]>> I fear that I'm missing your point.


How is
l[[0, -1]] or fname, lname = d[['first_name', 'last_name']]
any better than
l[ 0 ], l[ -1 ] or
fname = d[ 'first_name' ]
lname = d[ 'last_name' ]


It's more compact, especially, if "d" isn't a one-character variable,
but an expression:

 fname, lname = db[people].employee.object.get(pk=1234)[['first_name', 
'last_name']]

vs.

 fname = db[people].employee.object.get(pk=1234)['first_name']
 lname = db[people].employee.object.get(pk=1234)['last_name']


I have to say I don't think that's more compact at all.  It's too wide 
to be compact.  I think the second version is more readable (and the 
third version, where you factored out the common lookup is better still).


--
Rhodri James *-* Kynesim Ltd
--
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Peter J. Holzer]

On 2020-03-19 18:22:34 +1300, DL Neil via Python-list wrote:
> On 19/03/20 3:28 PM, Santiago Basulto wrote:
> > myself missing A LOT features from NumPy, like fancy indexing or
> > boolean arrays.
> > So, has it ever been considered to bake into Python's builtin list and
> > dictionary types functionality inspired by NumPy? I think multi indexing
> > alone would be huge addition. A few examples:
> > For lists and tuples:
> >  >>> l = ['a', 'b', 'c']
> >  >>> l[[0, -1]]
> >  ['a', 'c']
> > For dictionaries it'd even be more useful:
> >  d = {
> >  'first_name': 'Frances',
> >  'last_name': 'Allen',
> >  'email': 'fal...@ibm.com'
> >  }
> >  fname, lname = d[['first_name', 'last_name']]
> 
> 
> I fear that I'm missing your point.
> 
> How is
>   l[[0, -1]] or fname, lname = d[['first_name', 'last_name']]
> any better than
>   l[ 0 ], l[ -1 ] or
>   fname = d[ 'first_name' ]
>   lname = d[ 'last_name' ]

It's more compact, especially, if "d" isn't a one-character variable,
but an expression:

fname, lname = db[people].employee.object.get(pk=1234)[['first_name', 
'last_name']]

vs.

fname = db[people].employee.object.get(pk=1234)['first_name']
lname = db[people].employee.object.get(pk=1234)['last_name']

Plus the latter probably performs two database lookups, so you would
want to write:

person = db[people].employee.object.get(pk=1234)
fname = person['first_name']
lname = person['last_name']

(This is one of the Perl features I missed when I started using Python)

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[DL Neil via Python-list]

On 19/03/20 3:28 PM, Santiago Basulto wrote:
...> myself missing A LOT features from NumPy, like fancy indexing or 
boolean

arrays.
So, has it ever been considered to bake into Python's builtin list and
dictionary types functionality inspired by NumPy? I think multi indexing
alone would be huge addition. A few examples:
For lists and tuples:
 >>> l = ['a', 'b', 'c']
 >>> l[[0, -1]]
 ['a', 'c']
For dictionaries it'd even be more useful:
 d = {
 'first_name': 'Frances',
 'last_name': 'Allen',
 'email': 'fal...@ibm.com'
 }
 fname, lname = d[['first_name', 'last_name']]



I fear that I'm missing your point.

How is
l[[0, -1]] or fname, lname = d[['first_name', 'last_name']]
any better than
l[ 0 ], l[ -1 ] or
fname = d[ 'first_name' ]
lname = d[ 'last_name' ]

Are you aware, that whilst there is more coverage of "tuple unpacking" 
(at least to my eye), there is also a "list unpacking" feature?


>>> t = ( 1, 2, 3 )
>>> a, b, c = t
>>> print( a, b, c )
1 2 3

>>> l = [ 1, 2, 3 ]
>>> a, b, c = l
>>> print( a, b, c )
1 2 3

and somewhat similarly for dictionaries:

>>> fname, lname = d[ "first_name" ], d[ "last_name" ]
>>> fname, lname
('Frances', 'Allen')


That said, I've often wished to be allowed to write:

d.first_name

for a dict (cf a class/object).

Hmm, I feel a 'utility' coming-on - but first I'll look to see where/how 
such might be used in 'live' code (and be any better than the current 
mechanisms)...


Simple collections are one thing. How would you handle the structure if 
one or more elements contains a second dimension? eg a list within a 
list/a 2D matrix (or if you must, an 'array')?

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

PEP Idea: Multi-get for lists/tuples and dictionaries (inspired in NumPy)

[Santiago Basulto]

Hello community. I have an idea to share with the list to see what you all
think about it.

I happen to use both Python for Data Science (with our regular friends
NumPy and Pandas) as well as for scripting and backend development. Every
time I'm working in server-side Python (not the PyData stack), I find
myself missing A LOT features from NumPy, like fancy indexing or boolean
arrays.

So, has it ever been considered to bake into Python's builtin list and
dictionary types functionality inspired by NumPy? I think multi indexing
alone would be huge addition. A few examples:

For lists and tuples:
>>> l = ['a', 'b', 'c']
>>> l[[0, -1]]
['a', 'c']

For dictionaries it'd even be more useful:
d = {
'first_name': 'Frances',
'last_name': 'Allen',
'email': 'fal...@ibm.com'
}
fname, lname = d[['first_name', 'last_name']]

I really like the syntax of boolean arrays too, but considering we have
list comprehensions, seems a little more difficult to sell.

I'd love to see if this is something people would support, and see if
there's room to submit a PEP.

-- 
Santiago Basulto.-
Up!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Chris Angelico]

On Fri, Sep 20, 2019 at 5:16 AM Cecil Westerhof  wrote:
>
> Paul Rubin  writes:
>
> > Python 3.7.3 (default, Apr  3 2019, 05:39:12)
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> a = range(10)
> > >>> b = reversed(a)
> > >>> sum(a) == sum(b)
> > True
> > >>> sum(b) == sum(a)
> > False
>
> Why does this happen?
>
> By the way, when you change the last statement to:
> sum(a) == sum(b)
>
> you also get False.

>>> sum(range(10)) == sum(reversed(range(10)))
True

If you actually want a reversed range, use slice notation instead of
the reversed() function, which is more parallel to iter().

>>> a = range(10)
>>> b = a[::-1]
>>> sum(a) == sum(b)
True
>>> sum(b) == sum(a)
True

Now you actually have a range that runs the other direction, instead
of an iterator that runs through the range backwards.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Cecil Westerhof]

Paul Rubin  writes:

> Python 3.7.3 (default, Apr  3 2019, 05:39:12)
> Type "help", "copyright", "credits" or "license" for more information.
> >>> a = range(10)
> >>> b = reversed(a)
> >>> sum(a) == sum(b)
> True
> >>> sum(b) == sum(a)
> False

Why does this happen?

By the way, when you change the last statement to:
sum(a) == sum(b)

you also get False.

-- 
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: OT: Using a fake Gmail address is probably not a good idea

[Max Zettlmeißl via Python-list]

On Mon, Sep 16, 2019 at 1:56 PM Skip Montanaro  wrote:
> Mails for someone here who goes by the handle "ast" with a fake
> address of n...@gmail.com keep landing in my Gmail spam folder. I
> suspect the same is true for all people subscribed to python-list who
> use Gmail. Gmail (correctly, I think) can't verify that the mail I
> received actually originated there.

It is true for any server that is applying DMARC policies.
And having to deal with his messages is also very annoying to me.

Ast should use a proper invalid email address (E.g. anything ending in
.invalid, but there are also other reserved domains for such
purposes.) if he does not want to reveal his real address, instead of
making up a possibly valid address.
-- 
https://mail.python.org/mailman/listinfo/python-list

OT: Using a fake Gmail address is probably not a good idea

[Skip Montanaro]

(I would have sent this off-list, but for obvious reasons I couldn't.)

Mails for someone here who goes by the handle "ast" with a fake
address of n...@gmail.com keep landing in my Gmail spam folder. I
suspect the same is true for all people subscribed to python-list who
use Gmail. Gmail (correctly, I think) can't verify that the mail I
received actually originated there. It appears this user is actually
posting via Usenet through the server news.free.fr (apparently
ProxAd). I check my spam folder regularly for false positives, but I'm
sure many people don't. Whoever you are, ast, you might want to
reconsider your choice of fake return address.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Morten W. Petersen]

goto main;

Blogging at http://blogologue.com
Tweeting at https://twitter.com/blogologue
On Instagram https://instagram.com/morphexx

søn. 15. sep. 2019, 03.07 skrev Christian Seberino :

> Python is my goto main language.  However, sometimes I'm tempted to
> play with a Lisp like language just for fun.
>
> Clojure looks pretty solid but its syntax is different than Python's.
>
> Since Lisp's make it so easy to modify the language, what about the idea
> of developing a few macros to make a modified Clojure dialect that
> more closely matched Python keywords and so was more comfy for Pythonistas?
>
> Chris
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Michael Torrie]

On 9/15/19 9:10 AM, Christian Seberino wrote:
> Say if I may ask a tangential question...I've always wondered whether it 
> would be not too hard to compile Python source code to a Lisp like source 
> code?  How hard would it be to say compile Python source to Clojure source?

I'm sure a compiler could take a non-dynamic subset of Python and
transpile it into another language. Or you could implement an
interpreter in Clojure.

A project called Nuitka turns Python code into a compiled binary. I'm
not quite sure how it works since the dynamic parts of Python have to be
interpreted at runtime still.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Luciano Ramalho]

Take a look at this, Christian:

https://github.com/lihaoyi/macropy

Best,

Luciano

On Sun, Sep 15, 2019 at 12:17 PM Christian Seberino  wrote:
>
>
> > I had to read this twice.  It confused the hell out of me.
>
> Lol.  Certainly didn't mean to be confusing!  Hy bring Lisp to Python.  I was
> more interested in making a Lisp that had trivial similarities to Python like 
> using some of the same keywords.
>
> A related interested of mine is converting Python source code to Clojure 
> source.  Imagine instead compiling Python source to an intermediate Lisp 
> flavor that was closer to Python.  Then, later converting that "Pythonic 
> Lisp" to Clojure later.  That "Pythonic Lisp" is what I was thinking about.
>
> Cheers,
>
> Chris
>
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Christian Seberino]

> Python vs Clojure's syntax difference is superficial compared to their
> other differences, like the Clojure's immutable data structures and
> having to deal with the JVM.

Well there's ClojureScript to run this hypothetical Pythonic Lisp in the 
browser.

> I also don't think it's really practical
> to Pythonize Lisp syntax using macros as opposed to using Lisp as a
> compiler backend for something with a non-Lisp syntax.

Say if I may ask a tangential question...I've always wondered whether it would 
be not too hard to compile Python source code to a Lisp like source code?  How 
hard would it be to say compile Python source to Clojure source?

cs
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Christian Seberino]

> I had to read this twice.  It confused the hell out of me.

Lol.  Certainly didn't mean to be confusing!  Hy bring Lisp to Python.  I was
more interested in making a Lisp that had trivial similarities to Python like 
using some of the same keywords.

A related interested of mine is converting Python source code to Clojure 
source.  Imagine instead compiling Python source to an intermediate Lisp flavor 
that was closer to Python.  Then, later converting that "Pythonic Lisp" to 
Clojure later.  That "Pythonic Lisp" is what I was thinking about.

Cheers,

Chris

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Michael Torrie]

On 9/14/19 8:19 PM, Louis Valence wrote:
> I had to read this twice.  It confused the hell out of me.  Anyhow, I
> suppose you should take a look at
> 
>   https://github.com/hylang/hy

Yup that's probably exactly the opposite of what the OP was asking
about.  Neat, though.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Louis Valence]

Christian Seberino  writes:

> Python is my goto main language.  However, sometimes I'm tempted to
> play with a Lisp like language just for fun.
>
> Clojure looks pretty solid but its syntax is different than Python's.
>
> Since Lisp's make it so easy to modify the language, what about the idea
> of developing a few macros to make a modified Clojure dialect that
> more closely matched Python keywords and so was more comfy for Pythonistas?

I had to read this twice.  It confused the hell out of me.  Anyhow, I
suppose you should take a look at

  https://github.com/hylang/hy

Enjoy!
-- 
https://mail.python.org/mailman/listinfo/python-list

What about idea of making a "Pythonic Lisp"...i.e. a Lisp that more closely resembles the syntax of Python?

[Christian Seberino]

Python is my goto main language.  However, sometimes I'm tempted to
play with a Lisp like language just for fun.

Clojure looks pretty solid but its syntax is different than Python's.

Since Lisp's make it so easy to modify the language, what about the idea
of developing a few macros to make a modified Clojure dialect that
more closely matched Python keywords and so was more comfy for Pythonistas?

Chris
-- 
https://mail.python.org/mailman/listinfo/python-list

(yet another) PEP idea to tackle binary wheels problem efficiently

[Alexander Revin]

Hi all,

I've been thoroughly reading various discussions, such as [1], [2] and
related ones regarding PEP 425, PEP 491 and PEP 513. I also happen to
use musl sometimes, so as discussed here [3] I thought it would be a
good idea to submit a new PEP regarding musl compatibility.

It's not a secret that everyday wheel usage on Linux is far from
perfect. Some packages are trying to compile when there's no compiler
on the system available, some rely on 3rd-party deps and explode when
they cannot find required headers installed and so on. Add to this
cross-compilation support (quite complicated) and distros like Alpine
or just something not using x86 (how many piwheels.org-like should
emerge for each non-x86 platform?). For example, I would like to
seamlessly install numpy, pandas and scikit onto ppc machine running
Gentoo musl and not waste 1 hour for compilation, or "just" use them
in x86 standard alpine-based docker image (basically what [3] is all
about).

Long story short, current wheel filename format:

{distribution}-{version}(-{build tag})?-{python tag}-{abi
tag}-{platform tag}.whl.

Doesn't not properly express package expectation. Let's take a look at
pandas wheels ([4]):

pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl
pandas-0.24.1-cp36-cp36m-win_amd64.whl
pandas-0.24.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

I see issues with each of them:
1. First one won't work on Alpine or any musl-based distro;
2. Second – how amd64 is different from x86_64?
3. Third's platform tag is just a nightmare.

More of that, if you open the last one and inspect one of the
libraries, you'll find that:
$ file _libs/algos.cpython-36m-darwin.so
_libs/algos.cpython-36m-darwin.so: Mach-O universal binary with 2
architectures: [i386:Mach-O bundle i386] [x86_64]
_libs/algos.cpython-36m-darwin.so (for architecture i386): Mach-O bundle i386
_libs/algos.cpython-36m-darwin.so (for architecture x86_64): Mach-O
64-bit bundle x86_64

It's universal library! So not x86_64 only, as mentioned in the quite
long macosx_10_various platform tag.

TL;DR What my solution? To use something widely called "Target
Triplet" [5], omitting usual "vendor" field, so
{platform tag} from PEP 435 will have the format of _[_]:

pandas-0.24.1-cp36-cp36m-x86_64_linux_gnu.whl
pandas-0.24.1-cp36-cp36m-x86_64_linux_musl.whl
pandas-0.24.1-cp36-cp36m-x86_windows.whl
pandas-0.24.1-cp36-cp36m-x86_64_windows_msvs2010.whl
pandas-0.24.1-cp36-cp36m-x86_macosx_10_6.whl <-- won't be used for
anything Mojave+, see [6]
pandas-0.24.1-cp36-cp36m_aarch64_netbsd.whl

Primary concerns here:
- Arch and os are specified more consistently;
- Environment is specified only when needed;
- Lots of possible combinations of os and env are possible :)

Since most runtimes are hardcoded during build time anyway and changed
for each Python release, explicit versioning shouldn't be a problem.

JavaScript and Rustlang [7] use similar naming scheme. Though I don't
like both of them, at least portability of extensions looks less
broken than of Python (I've worked on native Nodejs extension in the
past).


What do you think?

Thanks,
Alex

[1] 
https://mail.python.org/archives/list/distutils-...@python.org/thread/KCLRIN4PTUGZLLL7GOUM23S46ZZ2D4FU/
[2] https://github.com/pypa/packaging-problems/issues/69
[3] https://github.com/pypa/manylinux/issues/37
[4] https://pypi.org/project/pandas/#files
[5] https://wiki.osdev.org/Target_Triplet
[6] https://support.apple.com/en-us/HT208436
[7] https://doc.rust-lang.org/reference/conditional-compilation.html
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: (yet another) PEP idea to tackle binary wheels problem efficiently

[Alexander Revin]

Two minor typos: platform tag should be separated by "-", not "_".
Also it makes sense to use "amd64" instead of "x86_64", so platform
can be just split by "_"

On Sat, Feb 16, 2019 at 9:29 PM Alexander Revin  wrote:
>
> Hi all,
>
> I've been thoroughly reading various discussions, such as [1], [2] and
> related ones regarding PEP 425, PEP 491 and PEP 513. I also happen to
> use musl sometimes, so as discussed here [3] I thought it would be a
> good idea to submit a new PEP regarding musl compatibility.
>
> It's not a secret that everyday wheel usage on Linux is far from
> perfect. Some packages are trying to compile when there's no compiler
> on the system available, some rely on 3rd-party deps and explode when
> they cannot find required headers installed and so on. Add to this
> cross-compilation support (quite complicated) and distros like Alpine
> or just something not using x86 (how many piwheels.org-like should
> emerge for each non-x86 platform?). For example, I would like to
> seamlessly install numpy, pandas and scikit onto ppc machine running
> Gentoo musl and not waste 1 hour for compilation, or "just" use them
> in x86 standard alpine-based docker image (basically what [3] is all
> about).
>
> Long story short, current wheel filename format:
>
> {distribution}-{version}(-{build tag})?-{python tag}-{abi
> tag}-{platform tag}.whl.
>
> Doesn't not properly express package expectation. Let's take a look at
> pandas wheels ([4]):
>
> pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl
> pandas-0.24.1-cp36-cp36m-win_amd64.whl
> pandas-0.24.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>
> I see issues with each of them:
> 1. First one won't work on Alpine or any musl-based distro;
> 2. Second – how amd64 is different from x86_64?
> 3. Third's platform tag is just a nightmare.
>
> More of that, if you open the last one and inspect one of the
> libraries, you'll find that:
> $ file _libs/algos.cpython-36m-darwin.so
> _libs/algos.cpython-36m-darwin.so: Mach-O universal binary with 2
> architectures: [i386:Mach-O bundle i386] [x86_64]
> _libs/algos.cpython-36m-darwin.so (for architecture i386): Mach-O bundle i386
> _libs/algos.cpython-36m-darwin.so (for architecture x86_64): Mach-O
> 64-bit bundle x86_64
>
> It's universal library! So not x86_64 only, as mentioned in the quite
> long macosx_10_various platform tag.
>
> TL;DR What my solution? To use something widely called "Target
> Triplet" [5], omitting usual "vendor" field, so
> {platform tag} from PEP 435 will have the format of _[_]:
>
> pandas-0.24.1-cp36-cp36m-x86_64_linux_gnu.whl
> pandas-0.24.1-cp36-cp36m-x86_64_linux_musl.whl
> pandas-0.24.1-cp36-cp36m-x86_windows.whl
> pandas-0.24.1-cp36-cp36m-x86_64_windows_msvs2010.whl
> pandas-0.24.1-cp36-cp36m-x86_macosx_10_6.whl <-- won't be used for
> anything Mojave+, see [6]
> pandas-0.24.1-cp36-cp36m_aarch64_netbsd.whl
>
> Primary concerns here:
> - Arch and os are specified more consistently;
> - Environment is specified only when needed;
> - Lots of possible combinations of os and env are possible :)
>
> Since most runtimes are hardcoded during build time anyway and changed
> for each Python release, explicit versioning shouldn't be a problem.
>
> JavaScript and Rustlang [7] use similar naming scheme. Though I don't
> like both of them, at least portability of extensions looks less
> broken than of Python (I've worked on native Nodejs extension in the
> past).
>
>
> What do you think?
>
> Thanks,
> Alex
>
> [1] 
> https://mail.python.org/archives/list/distutils-...@python.org/thread/KCLRIN4PTUGZLLL7GOUM23S46ZZ2D4FU/
> [2] https://github.com/pypa/packaging-problems/issues/69
> [3] https://github.com/pypa/manylinux/issues/37
> [4] https://pypi.org/project/pandas/#files
> [5] https://wiki.osdev.org/Target_Triplet
> [6] https://support.apple.com/en-us/HT208436
> [7] https://doc.rust-lang.org/reference/conditional-compilation.html
-- 
https://mail.python.org/mailman/listinfo/python-list

PyLint and Mypy real-time (and on-demand) code inspection from within PyCharm/IDEA

[Roberto Leinardi]

Hello there,

I am the developer of pylint-pycharm and mypy-pycharm, two plugins
providing both real-time and on-demand scanning of Python files with
PyLint/Mypy from within PyCharm/IDEA.

The real-time code inspection works automatically the same way like the
PyCharm's build-in PEP8 check (you see the issues highlighted directly in
your code while typing).

The on-demand inspection has several options that go from just scanning the
current file to scan the entire project.

The plugins also offer an option to run a check on modified files before a
VCS checkin.

If you are familiar with Checkstyle-IDEA plugin for Java, they are very
similar and offer the same features (and share a lot of the code).

The plugins source code is available here:
https://github.com/leinardi/pylint-pycharm
https://github.com/leinardi/mypy-pycharm

But they can also be easily installed from the official JetBrains Plugin
Repository:
1. In PyCharm, open the Settings/Preferences dialog (CTRL+Alt+S), click
Plugins.
2. Click Browse repositories.
3. In the Browse Repositories dialog that opens, right-click on the plugin
named "Pylint" or "Mypy" and select Download and Install.
4. Confirm your intention to download and install the selected plugin.
5. Click Close.
6. Click OK in the Settings dialog and restart PyCharm for the changes to
take effect.

Thanks and enjoy linting!
Roberto
-- 
https://mail.python.org/mailman/listinfo/python-list

I try to run selenium with firefox on RasberyPi3 ARM7 , any idea if it possible?

[alon . najman]

I try to run selenium with firefox on RasberyPi3 ARM7 , any idea if it possible?

what should I do?
I don't mind to change to another driver
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue29929] Idea: Make __main__ an implied package

[Nick Coghlan]

Nick Coghlan added the comment:

A key enabler for actually pursuing this idea would be coming up with a 
feasible way of emitting a deprecation warning for code that relied on the old 
implicit relative imports. A reasonable fast check for that would be to:

1. Start populating a private sys._main_path_entry in the sys module in 
addition to including it in both __main__.__path__ and sys.path

2. During the deprecation period, emit a warning when an import is satisfied 
from the sys._main_path_entry directory and the fully qualified module name 
*doesn't* start with "__main__."

3. After the deprecation period, stop populating sys.path[0], stop setting 
sys._main_path_entry, and stop emitting the deprecation warning

There's still plenty of details to be worked out before this idea could become 
reality (especially in terms of how it relates to module execution with the -m 
switch), but both the idea and a managed migration away from the status quo 
seem like they should be feasible.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29929>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29929] Idea: Make __main__ an implied package

[Nick Coghlan]

New submission from Nick Coghlan:

In just the last 24 hours, I've run across two cases where the default "the 
script directory is on sys.path" behaviour confused even experienced 
programmers:

1. a GitHub engineer thought the Python version in their Git-for-Windows bundle 
was broken because "from random import randint" failed (from a script called 
"random.py"

2. a Red Hat engineer was thoroughly confused when their systemd.py script was 
executed a second time when an unhandled exception was raised (Fedora's system 
Python is integrated with the ABRT crash reporter, and the except hook 
implementation does "from systemd import journal" while dealing with an 
unhandled exception)

This isn't a new problem, we've known for a long time that people are regularly 
confused by this, and it earned a mention as one of my "Traps for the Unwary in 
Python's Import System": 
http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-name-shadowing-trap

However, what's changed is that for the first time I think I see a potential 
way out of this: rather than injecting the script directory as sys.path[0], we 
could set it as "__main__.__path__ = []".

Cross-version compatible code would then be written as:

if "__path__" in globals():
from . import relative_module_name
else:
import relative_module_name

This approach would effectively be a continuation of PEP 328 (which eliminated 
implicit relative imports from within packages) and PEP 366 (which allowed 
implicit relative imports from modules executed with the '-m' switch).

--
components: Interpreter Core
messages: 290689
nosy: ncoghlan
priority: normal
severity: normal
status: open
title: Idea: Make __main__ an implied package
type: enhancement
versions: Python 3.7

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29929>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: What do you think: good idea to launch a marketplace on python+django?

[Gus_G]

W dniu piątek, 2 grudnia 2016 21:51:08 UTC+1 użytkownik codew...@gmail.com 
napisał:
 
> Something like this: https://marketplace.django-cms.org/en/ ?

Hey, that's nice solution, good to know that there is site like this :D Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: What do you think: good idea to launch a marketplace on python+django?

[codewizard]

On Friday, December 2, 2016 at 2:01:57 AM UTC-5, Gus_G wrote:
> Hello, what do you think about building a marketplace website on connection 
> of python+django? End effect-side should look and work similar to these: 
> https://zoptamo.com/uk/s-abs-c-uk, https://www.ownerdirect.com/ . What are 
> your opinions on this idea? Maybe there is other, better way to build it?

Something like this: https://marketplace.django-cms.org/en/ ?
-- 
https://mail.python.org/mailman/listinfo/python-list

What do you think: good idea to launch a marketplace on python+django?

[Gus_G]

Hello, what do you think about building a marketplace website on connection of 
python+django? End effect-side should look and work similar to these: 
https://zoptamo.com/uk/s-abs-c-uk, https://www.ownerdirect.com/ . What are your 
opinions on this idea? Maybe there is other, better way to build it?
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

Would be nice to add this, but I have no immediate plans so I'm unassigning it.

--
assignee: steve.dower -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Namespaces are one honking great idea

[carlosjosepita]

Hi all,

although it doesn't fit the bill 100%, I sometimes use this extremely simple 
function as a decorator:

def new(call):
return call()


For example:

@new
class MySingleton:
x = 2
y = 2
def sum(self, x, y):
return x + y


@new
def my_obj():
x = 2
y = 2
def sum(x, y):
return x + y
return Bundle(locals())


where Bundle is a simple subclass of dict implementing __xxxattr__ dunder 
methods.

Cheers
--
Carlos


 
-- 
https://mail.python.org/mailman/listinfo/python-list

A nestedmodule decorator (Re: Namespaces are one honking great idea)

[Gregory Ewing]

Steven D'Aprano wrote:

There's only so far I can go without support from the compiler.


It turns out one can go surprisingly far. Here's something I
cooked up that seems to meet almost all the requirements.
The only shortcoming I can think of is that a nestedmodule
inside another nestedmodule won't be able to see the names
in the outer nestedmodule directly (much like nested classes).

% python3 test_nestedmodule.py
0.7071067811865475

#--
#
#   test_nestedmodule.py
#
#--

from math import pi, sin
from nestedmodule import nestedmodule

def f(x):
return x**2

@nestedmodule
def test():

def g(x):
return f(x) * pi

def h(x):
return sin(g(x))

y = test.h(0.5)
print(y)

#--
#
#   nestedmodule.py
#
#--

from types import CodeType, ModuleType

def hack_code(f):
"""Hack 'return locals()' onto the end of the bytecode of f."""
code1 = f.__code__
bytes1 = code1.co_code
names1 = code1.co_names
n = len(names1)
names2 = names1 + ('locals',)
bytes2 = bytes1[:-4] + bytes([116, n, 0, 131, 0, 0, 83])
code2 = CodeType(code1.co_argcount, code1.co_kwonlyargcount, 
code1.co_nlocals,
code1.co_stacksize, code1.co_flags, bytes2, code1.co_consts, names2,
code1.co_varnames, code1.co_filename, code1.co_name, 
code1.co_firstlineno,
code1.co_lnotab, code1.co_freevars, code1.co_cellvars)
return code2

def nestedmodule(f):
c = hack_code(f)
l = eval(c, f.__globals__)
m = ModuleType(f.__name__)
m.__dict__.update(l)
return m
--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Tuesday 05 July 2016 13:47, Chris Angelico wrote:

> On Tue, Jul 5, 2016 at 1:35 PM, Steven D'Aprano  wrote:
>>> If you push your code into a separate .py file, you can reference the
>>> original module by importing it. Is that also the normal way to use
>>> "outer" functions etc from inside a namespace?
>>
>> Good question!
>>
>> With the current implementation, importing should work, but it's not
>> necessary. The surrounding module (the real .py module) is inserted into
>> the name resolution path of functions:
>>
>> py> x = 999
>> py> @namespace.Namespace
>> ... class Test:
>> ... def test():
>> ... print(x)
>> ...
>> py> Test.test()
>> 999
> 
> Ah, fascinating. This does break the "just unindent and move to a new
> file if you want to break it out" equivalency, but it does make sense
> - it's a *nested* namespace, which modules (even in a package) are
> not. So you have the outer namespace acting pretty much the way
> builtins do. (Do nested namespaces work?)

I haven't got that far, but I expect that nested namespaces will be nested in 
name only, like nested functions in Python 1.5.

There's only so far I can go without support from the compiler. If you nest a 
class inside another class, the inner class doesn't see variables in the outer 
class even during construction. So I'm pretty sure my namespace metaclass will 
inherit the same limitation.



-- 
Steve

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Chris Angelico]

On Tue, Jul 5, 2016 at 1:35 PM, Steven D'Aprano  wrote:
>> If you push your code into a separate .py file, you can reference the
>> original module by importing it. Is that also the normal way to use
>> "outer" functions etc from inside a namespace?
>
> Good question!
>
> With the current implementation, importing should work, but it's not
> necessary. The surrounding module (the real .py module) is inserted into
> the name resolution path of functions:
>
> py> x = 999
> py> @namespace.Namespace
> ... class Test:
> ... def test():
> ... print(x)
> ...
> py> Test.test()
> 999

Ah, fascinating. This does break the "just unindent and move to a new
file if you want to break it out" equivalency, but it does make sense
- it's a *nested* namespace, which modules (even in a package) are
not. So you have the outer namespace acting pretty much the way
builtins do. (Do nested namespaces work?)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Tue, 5 Jul 2016 12:58 pm, Chris Angelico wrote:

> On Tue, Jul 5, 2016 at 12:34 PM, Steven D'Aprano 
> wrote:
>> *** IF *** you are willing to push the code out into its own separate .py
>> file, you can use a module and write your code in a more natural form:
>>
>>
>> # module example.py
>> var = 999
>>
>> def spam(arg):
>> return eggs(arg) + var
>>
>> def eggs(arg):
>> return arg*2
>>
>>
>> What I'm calling a "namespace" is just a module object that lives inside
>> another module, without requiring a separate .py file. It only uses
>> the "class" statement for pragmatic reasons: there's no other statement
>> available that will do the job.
> 
> If you push your code into a separate .py file, you can reference the
> original module by importing it. Is that also the normal way to use
> "outer" functions etc from inside a namespace?

Good question! 

With the current implementation, importing should work, but it's not
necessary. The surrounding module (the real .py module) is inserted into
the name resolution path of functions:

py> x = 999
py> @namespace.Namespace
... class Test:
... def test():
... print(x)
...
py> Test.test()
999


Of course, the module-global x will be shadowed by any x in the Test
namespace (which is the intent), and you cannot assign to them (also a
feature). A bare `x = 1` inside the function will make x a local, unless
you declare it global first, in which case it should assign to the Test
namespace scope instead.




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Chris Angelico]

On Tue, Jul 5, 2016 at 12:34 PM, Steven D'Aprano  wrote:
> *** IF *** you are willing to push the code out into its own separate .py
> file, you can use a module and write your code in a more natural form:
>
>
> # module example.py
> var = 999
>
> def spam(arg):
> return eggs(arg) + var
>
> def eggs(arg):
> return arg*2
>
>
> What I'm calling a "namespace" is just a module object that lives inside
> another module, without requiring a separate .py file. It only uses
> the "class" statement for pragmatic reasons: there's no other statement
> available that will do the job.

If you push your code into a separate .py file, you can reference the
original module by importing it. Is that also the normal way to use
"outer" functions etc from inside a namespace?

# demo.py
pi = 3.14
def stupidfib(x):
if x < 2: return x
return stupidfib(x-1) + stupidfib(x-2)


Namespace asdf: # (or class, however it's done)
def foo(x):
return stupidfib(x * pi) / pi

How should foo reference those "even more global" names? "from .
import pi, stupidfib" would work if you converted the module into a
package ("mv demo.py demo/__init__.py"), and "from demo import pi,
stupidfib" would work if you converted the namespace into a peer
module. Either could make sense.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Mon, 4 Jul 2016 09:23 pm, jmp wrote:

> On 07/01/2016 04:13 PM, Steven D'Aprano wrote:
>> But classes are not like the others: they must be instantiated before
>> they can be used, and they are more than just a mere namespace grouping
>> related entities. Classes support inheritance. Classes should be used for
>> "is-a" relationships, not "has-a" relationships. Although classes (and
>> instances) are namespaces, they provide fundamentally different kind of
>> behaviour than modules and packages.
> 
> A namespace would not hurt but I really don't get why you don't consider
> classes a valid and rather helpful namespace.

I never said that.

This is where the term "namespace" can be ambiguous. "Namespace" can refer
to any of:

- an abstract mapping of symbols (names) to values;

- specific kinds of namespaces:

  * the concrete C++/C#/PHP data type called "namespace";
  * Python packages and modules;
  * classes;
  * instances of a class;

- the implementation (the __dict__ attribute of modules, classes);

etc. Now clearly a class is not the same thing as the class __dict__, and a
module is not the same as a class, and neither modules nor classes are the
same as a C++ namespace. Doesn't mean that classes aren't valid namespaces,
just that their semantics, use-cases and behaviour are different to those
of modules.



> 1/ classes do not have to be instantiated.

That's a bit of a sore point to me.

Some years ago I wrote here to ask what name is given to a class that is not
instantiated before being used. Singleton classes get instantiated once,
allowing a single instance. What if you had a class that didn't need
instantiating at all, so that the class itself was *effectively* the
singleton? What should that be called?


Instead of this:

class SingletonClass:
...

singleton = SingletonClass()
singleton.method()


what if we had this instead?

class singleton:
...

singleton.method()


I was roundly told that this was crazy talk, that the whole point of classes
was that they must be instantiated to use them, that code like the second
example would be confusing and very possibly bring about the fall of
Western Civilisation. The idea that this might be a legitimate alternative
to the singleton design pattern was dismissed.

(The Python community is terribly conservative when it comes to code.)

And, in a sense, they were right: there are two common ways to get
singleton-like behaviour in general, and in Python specifically:

- use class that allows only a single instance;

- use a module.

Using the class itself is unusual and surprising (especially to Java
programmers, where classes aren't even first-class values), and more so,
it's *inconvenient*.

To write a class which is used without instantiation, you should raise an
error on instantiation, decorate every method using classmethod or
staticmethod, and have methods have to call each other using the dotted
name:

class Example:
var = 999

def __init__(self):
raise TypeError('do not instantiate this class')

@classmethod
def spam(cls, arg):
return cls.eggs(arg) + cls.var

@classmethod
def eggs(cls, arg):
return arg*2


*** IF *** you are willing to push the code out into its own separate .py
file, you can use a module and write your code in a more natural form:


# module example.py
var = 999

def spam(arg):
return eggs(arg) + var

def eggs(arg):
return arg*2


What I'm calling a "namespace" is just a module object that lives inside
another module, without requiring a separate .py file. It only uses
the "class" statement for pragmatic reasons: there's no other statement
available that will do the job.


> 2/ the fact that classes are more than a namespace is not an argument.
> Almost any object in python is able to do more than what you are
> actually using.

There's one aspect of classes that is a deliberate design feature, but goes
against what I'm after: the fact that the class namespace itself is NOT
part of the method name resolution rules (except during class
construction). Try this:

x = 999

class Example:
x = 1
print(x)  # called during class construction
@classmethod
def test(cls):
print(x)


Example.test()



This will print 1, then 999. We quite often get people complaining about
this. I'm not one of them. I want classes to keep the current rules. But it
shows that a class is not the right answer for a module-inside-a-module
object.


> 3/ Classes are used as much as 'is-a' than 'has-a', class instances
> *have* a state usually described by attributes

Instances have state, of course, but the relationship I'm talking about is
instance to class.

class Dog:
...

lassie = Dog()

Lassie is a dog, not "Lassie has a dog".

Lassie has a tail, not "Lassie is a tail".

Re: Namespaces are one honking great idea

[Lawrence D’Oliveiro]

On Tuesday, July 5, 2016 at 10:10:04 AM UTC+12, I wrote:
>
> On Monday, July 4, 2016 at 11:37:44 PM UTC+12, Chris Angelico wrote:
> 
> > Functions within the namespace can't call other functions within the
> > same namespace using unqualified names. This was a stated requirement.
> 
> Doesn’t my @namespace decorator provide that?

No it doesn’t.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 11:37:44 PM UTC+12, Chris Angelico wrote:

> Functions within the namespace can't call other functions within the
> same namespace using unqualified names. This was a stated requirement.

Doesn’t my @namespace decorator provide that?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[jmp]

On 07/04/2016 01:37 PM, Chris Angelico wrote:

On Mon, Jul 4, 2016 at 9:23 PM, jmp  wrote:

On 07/01/2016 04:13 PM, Steven D'Aprano wrote:


But classes are not like the others: they must be instantiated before they
can be used, and they are more than just a mere namespace grouping related
entities. Classes support inheritance. Classes should be used for "is-a"
relationships, not "has-a" relationships. Although classes (and instances)
are namespaces, they provide fundamentally different kind of behaviour
than
modules and packages.



A namespace would not hurt but I really don't get why you don't consider
classes a valid and rather helpful namespace.

1/ classes do not have to be instantiated.
2/ the fact that classes are more than a namespace is not an argument.
Almost any object in python is able to do more than what you are actually
using.
3/ Classes are used as much as 'is-a' than 'has-a', class instances *have* a
state usually described by attributes
4/ "Although classes (and instances) are namespaces, ". You seem to
contradict yourself. It was probably a rhetorical construct but it's rather
confusing.


Functions within the namespace can't call other functions within the
same namespace using unqualified names. This was a stated requirement.

ChrisA



Ho, I missed that one.

But if it's the only missing requirement, wouldn't be like stating that 
python instances are not instances because methods cannot call other 
methods without "self."ed qualified name ? We like explicit qualified 
stuff in python right ? ("explicit is better than implicit")


jm



--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Chris Angelico]

On Mon, Jul 4, 2016 at 9:23 PM, jmp  wrote:
> On 07/01/2016 04:13 PM, Steven D'Aprano wrote:
>>
>> But classes are not like the others: they must be instantiated before they
>> can be used, and they are more than just a mere namespace grouping related
>> entities. Classes support inheritance. Classes should be used for "is-a"
>> relationships, not "has-a" relationships. Although classes (and instances)
>> are namespaces, they provide fundamentally different kind of behaviour
>> than
>> modules and packages.
>
>
> A namespace would not hurt but I really don't get why you don't consider
> classes a valid and rather helpful namespace.
>
> 1/ classes do not have to be instantiated.
> 2/ the fact that classes are more than a namespace is not an argument.
> Almost any object in python is able to do more than what you are actually
> using.
> 3/ Classes are used as much as 'is-a' than 'has-a', class instances *have* a
> state usually described by attributes
> 4/ "Although classes (and instances) are namespaces, ". You seem to
> contradict yourself. It was probably a rhetorical construct but it's rather
> confusing.

Functions within the namespace can't call other functions within the
same namespace using unqualified names. This was a stated requirement.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[jmp]

On 07/01/2016 04:13 PM, Steven D'Aprano wrote:

But classes are not like the others: they must be instantiated before they
can be used, and they are more than just a mere namespace grouping related
entities. Classes support inheritance. Classes should be used for "is-a"
relationships, not "has-a" relationships. Although classes (and instances)
are namespaces, they provide fundamentally different kind of behaviour than
modules and packages.


A namespace would not hurt but I really don't get why you don't consider 
classes a valid and rather helpful namespace.


1/ classes do not have to be instantiated.
2/ the fact that classes are more than a namespace is not an argument. 
Almost any object in python is able to do more than what you are 
actually using.
3/ Classes are used as much as 'is-a' than 'has-a', class instances 
*have* a state usually described by attributes
4/ "Although classes (and instances) are namespaces, ". You seem to 
contradict yourself. It was probably a rhetorical construct but it's 
rather confusing.


jm

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Ethan Furman]

On 07/03/2016 03:02 PM, Kevin Conway wrote:
>At some point earlier Ethan Furman declared:


It's not a language change.


Perhaps. My argument is that anything that introduces a new class-like
construct and set of lexical scoping rules is a language change. For
example, if this change went into 2.7.13 would Jython suddenly be broken
because it hasn't implemented the new scoping rules?


It's not a language change*.  There is nothing for Jython, IronPython, 
Brython, etc., to implement.  No scoping rule changes, nothing.  The 
magic in Steven's name space is implemented by the metaclass by 
(presumably) rebuilding all the functions -- and that is how he manages 
the effective scoping rules.


--
~Ethan~


*Okay, it is not a language change the same way the addition of Enum was 
not a language change.  On the other hand, asyncio did have some 
language changes (await, async, etc.).

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[BartC]

On 04/07/2016 00:21, Lawrence D’Oliveiro wrote:

On Monday, July 4, 2016 at 10:02:59 AM UTC+12, Kevin Conway wrote:

If the problem with using classes to satisfy the namespace need is
that it's unwieldy to use dot qualified paths then isn't that quite similar
to saying namespaces are unwieldy?


Python has a simple solution to that:

a = mod.a
b = mod.b


Except that then

 a = 0

doesn't do the same thing as:

 mod.a = 0

But it'll probably work for things that are not assigned to.

--
Bartc

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Lawrence D’Oliveiro]

On Monday, July 4, 2016 at 10:02:59 AM UTC+12, Kevin Conway wrote:
> If the problem with using classes to satisfy the namespace need is
> that it's unwieldy to use dot qualified paths then isn't that quite similar
> to saying namespaces are unwieldy?

Python has a simple solution to that:

a = mod.a
b = mod.b

etc
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Kevin Conway]

>> Regardless, all use cases you've listed are already satisfied by use of
>> the static and class method decorators. Methods decorated with these do
>> not require an instance initialization to use.

> And are significantly less easy to use, as the functions MUST refer to
each
> other by their dotted names.

My response to this may come off as a bit glib, but it isn't intended that
way. If the problem with using classes to satisfy the namespace need is
that it's unwieldy to use dot qualified paths then isn't that quite similar
to saying namespaces are unwieldy? Leveraging classes as a nested module
creates a de-facto internal namespace of "cls" for self reference and I'm
unsure of why that is unwanted but "module.namespace.blah" is wanted.

I suppose my issue is not so much that namespace objects are a bad idea as
that the proposal does little to express why the existing tools are
deficient enough to require a new concept.

> For the proponents of namespace, what is deficient in the above example
> that necessitates a language change?

>> It's not a language change.

Perhaps. My argument is that anything that introduces a new class-like
construct and set of lexical scoping rules is a language change. For
example, if this change went into 2.7.13 would Jython suddenly be broken
because it hasn't implemented the new scoping rules?

>> IIRC, classes came about as a "module in a module".

> I'm pretty sure they did not. Object oriented programming (and hence
> classes) came about from simulating real world objects, hence the
> name:

I realize my statement was ambiguous. I didn't mean to suggest that
classes, as an idea and OOP tool, were derived from the concept of a
namespace. I meant to say that the Python implementation of classes is
quite similar to the implementation of modules in the cPython code. The
commit messages from the earlier days (ref: ab5db02) don't carry much
detail so I'm mostly basing my belief of the implementation similarities
and anecdotes from the community. Right or wrong, I don't believe it has
much weight in the namespace discussion and I shouldn't have brought it up.

On Sun, Jul 3, 2016, 00:17 Ethan Furman <et...@stoneleaf.us> wrote:

> On 07/02/2016 08:44 PM, Steven D'Aprano wrote:
>
> > Try getting this behaviour from within a class:
> >
> >
> > class Food(metaclass=Namespace):
> >
> >  # (1) no special decorators required
> >  def spam(n):
> >  return ' '.join(['spam']*n)
> >
> >  # (2) can call functions from inside the namespace
> >  breakfast = spam(5)
> >
> >  # (3) no "cls" or "self" argument
> >  def lunch():
> >  # (4) can access variables using their undotted name
> >  return breakfast + ' and eggs'
> >
> >  def supper():
> >  # (5) likewise functions (a special case of #4)
> >  return lunch() + ' and a fried slice of spam'
> >
> >  def mutate(n):
> >  # global inside the namespace refers to the namespace,
> >  # not the surrounding module
> >  global breakfast
> >  breakfast = spam(5)
>
> > Can you do all of that with an ordinary class?
>
> You can get #2 already, but not the rest (without your spiffy code ;) :
>
> Python 3.5.1+ (3.5:f840608f79da, Apr 14 2016, 12:29:06)
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> class Huh:
> ...   def blah(text):
> ... print('blah blah %s blah blah blah' % text)
> ...   blah('whatever')
> ...
> blah blah whatever blah blah blah
>
> --
> ~Ethan~
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Ethan Furman]

On 07/02/2016 08:44 PM, Steven D'Aprano wrote:


Try getting this behaviour from within a class:


class Food(metaclass=Namespace):

 # (1) no special decorators required
 def spam(n):
 return ' '.join(['spam']*n)

 # (2) can call functions from inside the namespace
 breakfast = spam(5)

 # (3) no "cls" or "self" argument
 def lunch():
 # (4) can access variables using their undotted name
 return breakfast + ' and eggs'

 def supper():
 # (5) likewise functions (a special case of #4)
 return lunch() + ' and a fried slice of spam'

 def mutate(n):
 # global inside the namespace refers to the namespace,
 # not the surrounding module
 global breakfast
 breakfast = spam(5)



Can you do all of that with an ordinary class?


You can get #2 already, but not the rest (without your spiffy code ;) :

Python 3.5.1+ (3.5:f840608f79da, Apr 14 2016, 12:29:06)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Huh:
...   def blah(text):
... print('blah blah %s blah blah blah' % text)
...   blah('whatever')
...
blah blah whatever blah blah blah

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Sun, 3 Jul 2016 01:34 am, Kevin Conway wrote:

> staticmethod isn't technically required to use a method through the class
> (or subclasses), it simply provides the appropriate magic to allow it to
> be called through instances.
> 
> For example, the following code covers all described use cases of the
> proposed namespace. 

It really doesn't. Here's your class:

> Methods are invoked without creating instances and 
> state is managed on the class object directly.
> 
> class CustomNs:
> 
> stateful_data = 1
> 
> @staticmethod
> def echo(text):
> print(text)
> 
> @classmethod
> def mutate(cls):
> cls.stateful_data += 1
> print(cls.stateful_data)


The class scope is not "first class" (pun intended), as you cannot refer to
class-level variables without prefixing them with the class, even from
within the class.

Although the class *may* be used without instantiating, this violates the
user's expectations and may be confusing.

I acknowledge that my own namespace has a similar problem, as it uses the
class keyword, but the use of an explicit Namespace decorator or metaclass
gives a hint that something is different. And if you try calling the
Namespace, as if to instantiate it, you get a TypeError.


[...]
> For the proponents of namespace, what is deficient in the above example
> that necessitates a language change?

It's not a language change. Although I'd like a new keyword, to avoid
the "but it looks like a class" problem, its not necessary.


Try getting this behaviour from within a class:


class Food(metaclass=Namespace):

# (1) no special decorators required
def spam(n):
return ' '.join(['spam']*n)

# (2) can call functions from inside the namespace
breakfast = spam(5)

# (3) no "cls" or "self" argument
def lunch():
# (4) can access variables using their undotted name
return breakfast + ' and eggs'

def supper():
# (5) likewise functions (a special case of #4)
return lunch() + ' and a fried slice of spam'

def mutate(n):
# global inside the namespace refers to the namespace,
# not the surrounding module
global breakfast
breakfast = spam(5)



Everything written inside the namespace object could have started as top
level module code. I select the code, hit my editor's "Indent" command, and
insert a single line at the top to turn it into a namespace.

If I decide to move the code into a separate file, I just copy the block,
excluding the "class ... metaclass" header line, past into a new file,
select all, and Dedent. Done.

Can you do all of that with an ordinary class?





-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Sat, 2 Jul 2016 11:50 am, Kevin Conway wrote:

> I believe the namespace object you are referring to is exactly a class.

Yes, a namespace is exactly like a class, minus inheritance, instantiation,
the implied "is-a" relationship, and the whole Java "utility class"
anti-pattern.

In other words, exactly not like a class *wink*

Classes and modules are both namespaces: "an abstract container or
environment created to hold a logical grouping of unique identifiers or
symbols (i.e. names)", to quote Wikipedia:

https://en.wikipedia.org/wiki/Namespace

But classes provide much more functionality, over and above mere namespace:
inheritance, instantiation, etc. I'm not anti-classes or opposed to OOP,
these features are great when they are wanted. But when they aren't wanted,
they're a nuisance.

Modules also provide more than just an abstract namespace: they provide
scope, but that's exactly what I want for my Namespace entity: it should be
a namespace, and it should be a scope for functions. In other words, it
should be a module. I just don't want to have to place the source code in a
separate file, because sometimes that's annoying and overkill.

Modules are good for information hiding and encapsulation, just like
classes, but without inheritance etc. But because you cannot put more than
one module inside a single file, the developer has to make a choice between
information hiding and encapsulation:

(1) You can keep your functions encapsulated inside a single module/file,
but you cannot hide some of them from parts of the module that don't care
about them.

(2) You can hide those functions from the parts of the module that don't
care about them, but only by breaking encapsulation, moving them to another
file, and exposing another name in the file system namespace.


The "namespace" feature offers an alternative.

Namespaces in C#, C++ and PHP:

https://msdn.microsoft.com/en-us/library/z2kcy19k.aspx
https://msdn.microsoft.com/en-us/library/5cb46ksf.aspx
http://php.net/manual/en/language.namespaces.php



> IIRC, classes came about as a "module in a module".

I'm pretty sure they did not. Object oriented programming (and hence
classes) came about from simulating real world objects, hence the name:

http://www.exforsys.com/tutorials/oops-concepts/the-history-of-object-oriented-programming.html

In languages like Pascal, you can nest functions and procedures inside
others, providing something like a namespace functionality, except that
Pascal strictly enforces information hiding. There's no way for code to
peer inside a Pascal function and see nested functions.


> Regardless, all use cases you've listed are already satisfied by use of
> the static and class method decorators. Methods decorated with these do
> not require an instance initialization to use.

And are significantly less easy to use, as the functions MUST refer to each
other by their dotted names.




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Ethan Furman]

On 07/02/2016 08:34 AM, Kevin Conway wrote:


For the proponents of namespace, what is deficient in the above example
that necessitates a language change?


Adding a new widget is not changing the language.

--
~Ethan~

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Kevin Conway]

> staticmethod isn't technically required to use a method through the class
(or subclasses), it simply provides the appropriate magic to allow it to be
called through instances.

For example, the following code covers all described use cases of the
proposed namespace. Methods are invoked without creating instances and
state is managed on the class object directly.

class CustomNs:

stateful_data = 1

@staticmethod
def echo(text):
print(text)

@classmethod
def mutate(cls):
cls.stateful_data += 1
print(cls.stateful_data)

CustomNs.echo("test")
print(CustomNs.stateful_data)
CustomNs.mutate()
print(CustomNs.stateful_data)

For the proponents of namespace, what is deficient in the above example
that necessitates a language change?

On Sat, Jul 2, 2016, 00:02 Random832  wrote:

> On Fri, Jul 1, 2016, at 21:50, Kevin Conway wrote:
> > I believe the namespace object you are referring to is exactly a
> > class. IIRC, classes came about as a "module in a module".
>
> No, because classes have instances. And conceptually they seem like they
> *should* have instances. Just using the term "class" carries
> expectations.
>
> More to the point, what modules do that classes do not is provide a
> global namespace for functions defined within them, so that variables
> within them can be used (well, read - writing them requires a
> declaration) by the functions without extra qualification.
>
> > Regardless, all use cases you've listed are already satisfied by use
> > of the static and class method decorators. Methods decorated with
> > these do not require an instance initialization to use.
>
> staticmethod isn't technically required to use a method through the
> class (or subclasses), it simply provides the appropriate magic to allow
> it to be called through instances.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Alexander Riccio]

Alexander Riccio added the comment:

We might want to use some kind of Group Policy setting, for the same reason 
that many Windows security configuration options are there, and that DoD STIGs 
for Windows https://www.stigviewer.com/stig/windows_8_8.1/ are almost totally 
about configuring Group Policy settings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Alexander Riccio]

Alexander Riccio added the comment:

It's not just Stuxnet, as at least one other Advanced Persistent Threat uses 
that tactic. An APT (likely Russian intelligence) recently used encoded 
PowerShell to break into the Democratic National Committe: 
https://www.crowdstrike.com/blog/bears-midst-intrusion-democratic-national-committee/

>From that article:

> This one-line powershell command, stored only in WMI database, establishes an 
> encrypted connection to C2 and downloads additional powershell modules from 
> it, executing them in memory.

(As a fun coincidence, they also used py2exe to distribute other modules, which 
is kinda like a separate interpreter using safe_exec)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Namespaces are one honking great idea

[Random832]

On Fri, Jul 1, 2016, at 21:50, Kevin Conway wrote:
> I believe the namespace object you are referring to is exactly a
> class. IIRC, classes came about as a "module in a module".

No, because classes have instances. And conceptually they seem like they
*should* have instances. Just using the term "class" carries
expectations.

More to the point, what modules do that classes do not is provide a
global namespace for functions defined within them, so that variables
within them can be used (well, read - writing them requires a
declaration) by the functions without extra qualification.

> Regardless, all use cases you've listed are already satisfied by use
> of the static and class method decorators. Methods decorated with
> these do not require an instance initialization to use.

staticmethod isn't technically required to use a method through the
class (or subclasses), it simply provides the appropriate magic to allow
it to be called through instances.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Lawrence D’Oliveiro]

On Saturday, July 2, 2016 at 1:50:56 PM UTC+12, Kevin Conway wrote:
> Regardless, all use cases you've listed are already satisfied by use of the
> static and class method decorators.

Except for the need to decorate every such function inside the class. How about:

import types

def namespace(inclass) :
"decorator which turns a class into a module."
outclass = types.ModuleType \
  (
name = inclass.__name__,
doc = inclass.__doc__,
  )
for attr in dir(inclass) :
if not attr.startswith("__") or attr in ("__package__",) :
setattr(outclass, attr, getattr(inclass, attr))
#end if
#end for
return \
outclass
#end namespace

Example use:

@namespace
class try_it :
"try it!"

val = "some text"

def func() :
print("func got called")
#end func

#end try_it
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Rustom Mody]

On Friday, July 1, 2016 at 8:19:36 PM UTC+5:30, BartC wrote:
> On 01/07/2016 15:13, Steven D'Aprano wrote:
> 
> > Sometimes we have a group of related functions and variables that belong
> > together, but are not sufficiently unrelated to the rest of the module that
> > we want to split them out into another file.
> 
> > Here's a proof of concept. I use a class with a custom metaclass like this:
> >
> >
> > # Python 3 version
> > class ExampleNS(metaclass=Namespace):
> > x = 1
> > y = []
> >
> > def spam(n):
> > return 'spam'*n
> 
> > py> Example.spam(5)
> > 'spamspamspamspamspam'
> 
> 
> Why not just extend the capabilities of a class? I actually thought this 
> would work until I tried it and it didn't:

Well I also thought similarly -- sure a normal (instance) method cannot be
used without an instance but sure this is what class/static methods are for?
ie Steven's option 4.

Then I remembered that python's LEGB rule is bizarre:
Outer scopes are added outside inner scopes ... except for classes
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Kevin Conway]

I believe the namespace object you are referring to is exactly a class.
IIRC, classes came about as a "module in a module".

Regardless, all use cases you've listed are already satisfied by use of the
static and class method decorators. Methods decorated with these do not
require an instance initialization to use.

On Fri, Jul 1, 2016, 20:17 Steven D'Aprano <st...@pearwood.info> wrote:

> On Sat, 2 Jul 2016 05:29 am, Ethan Furman wrote:
>
> > On 07/01/2016 10:10 AM, Steven D'Aprano wrote:
> >> On Sat, 2 Jul 2016 02:00 am, Ethan Furman wrote:
> >
> >>> Did you mean for this to go to -Ideas?
> >>
> >> Not yet. I wanted some initial feedback to see if anyone else liked the
> >> idea before taking it to Bikeshedding Central :-)
> >>
> >> Besides, I expect Python-Ideas will say it needs to be a package on PpPI
> >> first. Although I am kinda considering sneaking this into the std lib as
> >> an undocumented internal feature, like simplegeneric in pkgutil. (Don't
> >> tell anyone I said that *wink* )
> >
> > Are there good use-cases for this in the stdlib?
>
> I have at least one.
>
>
>
> --
> Steven
> “Cheer up,” they said, “things could be worse.” So I cheered up, and sure
> enough, things got worse.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Sat, 2 Jul 2016 05:29 am, Ethan Furman wrote:

> On 07/01/2016 10:10 AM, Steven D'Aprano wrote:
>> On Sat, 2 Jul 2016 02:00 am, Ethan Furman wrote:
> 
>>> Did you mean for this to go to -Ideas?
>>
>> Not yet. I wanted some initial feedback to see if anyone else liked the
>> idea before taking it to Bikeshedding Central :-)
>>
>> Besides, I expect Python-Ideas will say it needs to be a package on PpPI
>> first. Although I am kinda considering sneaking this into the std lib as
>> an undocumented internal feature, like simplegeneric in pkgutil. (Don't
>> tell anyone I said that *wink* )
> 
> Are there good use-cases for this in the stdlib?

I have at least one.



-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Ethan Furman]

On 07/01/2016 10:10 AM, Steven D'Aprano wrote:

On Sat, 2 Jul 2016 02:00 am, Ethan Furman wrote:



Did you mean for this to go to -Ideas?


Not yet. I wanted some initial feedback to see if anyone else liked the idea
before taking it to Bikeshedding Central :-)

Besides, I expect Python-Ideas will say it needs to be a package on PpPI
first. Although I am kinda considering sneaking this into the std lib as an
undocumented internal feature, like simplegeneric in pkgutil. (Don't tell
anyone I said that *wink* )


Are there good use-cases for this in the stdlib?

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Sat, 2 Jul 2016 12:49 am, BartC wrote:

> On 01/07/2016 15:13, Steven D'Aprano wrote:
> 
>> Sometimes we have a group of related functions and variables that belong
>> together, but are not sufficiently unrelated to the rest of the module
>> that we want to split them out into another file.
> 
>> Here's a proof of concept. I use a class with a custom metaclass like
>> this:
>>
>>
>> # Python 3 version
>> class ExampleNS(metaclass=Namespace):
>> x = 1
>> y = []
>>
>> def spam(n):
>> return 'spam'*n
> 
>> py> Example.spam(5)
>> 'spamspamspamspamspam'
> 
> 
> Why not just extend the capabilities of a class? 

The usual ways to change the behaviour of classes from Python code is via
decorator, which lets you modify the class after it is created, a
metaclass, which lets you modify it before it is created, or by using
descriptors to customize attribute access.

I'm using a metaclass. When you write:

class K: ...

that is syntactic sugar for calling the default metaclass (`type`) with some
arguments, and `type` then returns the new class K. But if you use a
metaclass (usually, but not necessarily a subclass of `type`) you can
customize the creation of the new class, or even return a completely
different object altogether. That's what I'm doing.

Why am I returning a module instead of a class? Because (1) conceptually a
namespace (in the C++ sense) is like a module; (2) classes have a whole lot
of expectations that aren't relevant to namespaces (like inheritance,
instantiation, "is-a", etc); and (3) it's easy to use a module, which
already provides most of the behaviour I want.


> I actually thought this 
> would work until I tried it and it didn't:
> 
> class C():
>  def fn():
>  print ("Hi!")
> 
> C.fn()

That actually will work in Python 3. (I had forgotten that.) In Python 2,
you have to inherit C from object, AND decorate fn with staticmethod:

class C(object):
@staticmethod
def fn():
print ("Hi!")


But for my use, that's not enough. I want this to work:

class Foo:
def f():
return g().upper()
def g():
return "hi!"


That is, Foo should behave as if it were a module imported from a file,
except without actually being imported from a file.




-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Steven D'Aprano]

On Sat, 2 Jul 2016 02:00 am, Ethan Furman wrote:

> On 07/01/2016 07:13 AM, Steven D'Aprano wrote:
> 
> I like the idea, but I have a couple questions about the design choices.

Thanks!

>   Comments below.


[...]
>> Despite the "class" statement (a limitation of Python's lack of dedicated
>> syntax for namespaces), the Example namespace behaves like (in fact,
>> *is*) a module embedded inside a module.
> 
> So the idea is to have several "mini-modules" inside a single file?

That would certainly be possible.


> Can a mini-module function access a different mini-module's function?

Yes, via the dotted attribute name, just as you might say "math.sin".

The only difference is that you don't use import to get access to the
(mini-)module. It's just there, defined in your own file.


> Can a mini-module function access/interact with the module's global scope?

Yes. I'm still experimenting with different implementations, but I have a
proof of concept working.


>> Especially note that inside the namespace, the global statement makes a
>> name refer to the namespace scope.
> 
> Interesting.  :)
> 
> 
>> Unresolved questions
>> =
>>
>> Would a decorator be better than a metaclass?
> 
> I think a decorator may provide more of a clue that something
> interesting is happening -- whether a decorator is feasible depends on
> whether you need the __prepare__ magic of metaclasses.

I don't think __prepare__ is needed.



>>  @namespace
>>  class Example:
>>  ...
> 
> I believe Python already has a NameSpace type, so a different name is
> probably better.  How about minimodule? ;)

I think you are thinking of "SimpleNamespace".

In any case, so long as my namespace lives in a different, er, namespace as
the other namespace, there's no conflict :-)

In the long term, I think my namespace (if accepted into the stdlib) would
belong in functools, collections or types. Perhaps even its own module. I
don't think this needs to be a built-in.



>> Some limitations
>> =
>>
>> The simplified implementation shown doesn't allow functions inside the
>> namespace to access names outside of the namespace easily.
> 
> How not-easy is it?

Not that hard now. I have an implementation working. It actually turned out
to be easy. (Thanks Raymond for ChainMap!)



>> In practice, the
>> user might want the name resolution order inside the namespace to be:
>>
>>   local variables
>>   nonlocal variables
>>   namespace globals
>>   module globals
>>   builtins
> 
> That would be ideal, I think.  How close can we get to that?

I now have that working, for some definition of working. Its not extensively
tested, so there might be some bugs or issues I haven't thought of, but the
basic concept is working: inside the namespace/mini-module object,
functions see namespace globals ahead of module globals, which are ahead of
builtins.


[...]
> Did you mean for this to go to -Ideas?


Not yet. I wanted some initial feedback to see if anyone else liked the idea
before taking it to Bikeshedding Central :-)

Besides, I expect Python-Ideas will say it needs to be a package on PpPI
first. Although I am kinda considering sneaking this into the std lib as an
undocumented internal feature, like simplegeneric in pkgutil. (Don't tell
anyone I said that *wink* )





-- 
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Chris Angelico]

On Sat, Jul 2, 2016 at 12:49 AM, BartC  wrote:
> Why not just extend the capabilities of a class? I actually thought this
> would work until I tried it and it didn't:
>
> class C():
> def fn():
> print ("Hi!")
>
> C.fn()
>
> The error message suggests Python knows what's going on. So why not just
> make it work?

rosuav@sikorsky:~$ python3
Python 3.6.0a2+ (default:57f3514564f6, Jun 29 2016, 16:27:34)
[GCC 5.3.1 20160528] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class C():
... def fn():
... print ("Hi!")
...
>>> C.fn()
Hi!
>>>

But if you have two such functions, they can't call each other without
the namespace marker.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Ethan Furman]

On 07/01/2016 07:13 AM, Steven D'Aprano wrote:

I like the idea, but I have a couple questions about the design choices. 
 Comments below.



The Zen of Python says:

 Namespaces are one honking great idea -- let's do more of those!



Proposal
=

Add a new "namespace" object to Python.




Proof of concept
=

Here's a proof of concept. I use a class with a custom metaclass like this:


# Python 3 version
class ExampleNS(metaclass=Namespace):
 x = 1
 y = []

 def spam(n):
 return 'spam'*n

 z = spam(3).upper()

 def ham(n):
 return spam(n).replace('sp', 'H')

 def test():
 global x
 x += 1
 y.append(1)
 return x, y




Despite the "class" statement (a limitation of Python's lack of dedicated
syntax for namespaces), the Example namespace behaves like (in fact, *is*)
a module embedded inside a module.


So the idea is to have several "mini-modules" inside a single file?

Can a mini-module function access a different mini-module's function?

Can a mini-module function access/interact with the module's global scope?


Especially note that inside the namespace, the global statement makes a name
refer to the namespace scope.


Interesting.  :)



Unresolved questions
=

Would a decorator be better than a metaclass?


I think a decorator may provide more of a clue that something 
interesting is happening -- whether a decorator is feasible depends on 
whether you need the __prepare__ magic of metaclasses.



 @namespace
 class Example:
 ...


I believe Python already has a NameSpace type, so a different name is 
probably better.  How about minimodule? ;)




Some limitations
=

The simplified implementation shown doesn't allow functions inside the
namespace to access names outside of the namespace easily.


How not-easy is it?


In practice, the
user might want the name resolution order inside the namespace to be:

  local variables
  nonlocal variables
  namespace globals
  module globals
  builtins


That would be ideal, I think.  How close can we get to that?


The biggest limitation is that I have to abuse the class statement to do
this. In an ideal world, there would be syntactic support and a keyword:

 namespace Example:
 x = 0
 y = []
 def test(n): ...

although others might argue that *not* needing a dedicated keyword is, in
fact, better.


Metaclasses for the win!


Disadvantages
==

Those unfamiliar with the namespace concept may be confused by a class that
doesn't get instantiated.


A well-named decorator/base-class should help with that.


Did you mean for this to go to -Ideas?

--
~Ethan~

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[BartC]

On 01/07/2016 15:13, Steven D'Aprano wrote:


Sometimes we have a group of related functions and variables that belong
together, but are not sufficiently unrelated to the rest of the module that
we want to split them out into another file.



Here's a proof of concept. I use a class with a custom metaclass like this:


# Python 3 version
class ExampleNS(metaclass=Namespace):
x = 1
y = []

def spam(n):
return 'spam'*n



py> Example.spam(5)
'spamspamspamspamspam'



Why not just extend the capabilities of a class? I actually thought this 
would work until I tried it and it didn't:


class C():
def fn():
print ("Hi!")

C.fn()

The error message suggests Python knows what's going on. So why not just 
make it work?


(The main issue would be, if an instance X of C /was/ created, then 
X.fn() wouldn't work as there is no 'self' parameter for fn.)


--
Bartc

--
https://mail.python.org/mailman/listinfo/python-list

Re: Namespaces are one honking great idea

[Random832]

On Fri, Jul 1, 2016, at 10:13, Steven D'Aprano wrote:
> The biggest limitation is that I have to abuse the class statement to do
> this. In an ideal world, there would be syntactic support and a keyword:
> 
> namespace Example:
> x = 0
> y = []
> def test(n): ...
> 
> although others might argue that *not* needing a dedicated keyword is, in
> fact, better.

What might be nice would be a single syntax that isn't specific to
classes or your "namespaces".

namespace(type) Name:
   "equivalent to class Name:"

namespace(whatever) Name:
   "equivalent to class Name(metaclass=whatever)"
-- 
https://mail.python.org/mailman/listinfo/python-list

Namespaces are one honking great idea

[Steven D'Aprano]

The Zen of Python says:

Namespaces are one honking great idea -- let's do more of those!



Proposal
=

Add a new "namespace" object to Python.


Rationale
==

Sometimes we have a group of related functions and variables that belong
together, but are not sufficiently unrelated to the rest of the module that
we want to split them out into another file.

Python allows us to group related chunks of code in three levels of
grouping:

- the class, containing one of more variable and/or function (method);
- the module, containing one or more classes and/or functions;
- the package, containing one or more modules.

But classes are not like the others: they must be instantiated before they
can be used, and they are more than just a mere namespace grouping related
entities. Classes support inheritance. Classes should be used for "is-a"
relationships, not "has-a" relationships. Although classes (and instances)
are namespaces, they provide fundamentally different kind of behaviour than
modules and packages.

Sometimes we have a group of related functions which belongs together, but
they are more closely related to each other than they are to the rest of
the module, but not so distantly unrelated that they should be split off
into a separate module. Nor do they belong as a class -- there's nothing
analogous to an instance, and no need for inheritance.

Perhaps the functions have some name clashes, e.g. two different functions
that deserve the same name, or perhaps they're just conceptually related.

What is needed is a module inside a module, but without the necessity of
refactoring the code into a package and splitting the code out into a
separate file. In other words, something rather like C++ namespaces:

https://msdn.microsoft.com/en-us/library/5cb46ksf.aspx


Proof of concept
=

Here's a proof of concept. I use a class with a custom metaclass like this:


# Python 3 version
class ExampleNS(metaclass=Namespace):
x = 1
y = []

def spam(n):
return 'spam'*n

z = spam(3).upper()

def ham(n):
return spam(n).replace('sp', 'H')

def test():
global x
x += 1
y.append(1)
return x, y


And a demonstration of its use:

py> Example.spam(5)
'spamspamspamspamspam'
py> Example.ham(5)
'HamHamHamHamHam'
py> Example.test()
(2, [1])
py> Example.test()
(3, [1, 1])
py> print(Example.x, Example.y, Example.z)
3 [1, 1] SPAMSPAMSPAM

Despite the "class" statement (a limitation of Python's lack of dedicated
syntax for namespaces), the Example namespace behaves like (in fact, *is*)
a module embedded inside a module.

Notice that the functions are not methods and don't require a "self"
parameter. Being functions, they can be used inside the body of the
namespace, including inside other functions.

Outside the namespace, access is via dotted attribute: Example.spam. But
inside it, functions can refer to each other without knowing the name of
their namespace.

Especially note that inside the namespace, the global statement makes a name
refer to the namespace scope.


Implementation
===

Here's a simplified implementation:


from types import FunctionType, ModuleType

class NamespaceType(ModuleType):
def __repr__(self):
return 'namespace <%s>' % self.__name__

class Namespace(type):
def __new__(meta, name, bases, ns):
if bases != ():
raise RuntimeError("namespaces cannot inherit")
module = NamespaceType(name)
globals_ = module.__dict__
if '__builtins__' not in globals_:
globals_['__builtins__'] = globals()['__builtins__']
for name, obj in ns.items():
if type(obj) is FunctionType:
ns[name] = meta._copy_and_inject(obj, globals_)
module.__dict__.update(ns)
return module

@classmethod
def _copy_and_inject(meta, func, globals):
"""Return a new function object with the given globals."""
assert type(func) is FunctionType
f = FunctionType(
func.__code__, globals, func.__name__,
 func.__defaults__, func.__closure__)
f.__dict__ = func.__dict__
# Next two lines are Python 3 only.
f.__kwdefaults__ = func.__kwdefaults__
f.__annotations__ = func.__annotations__
return f



The namespace object itself is actually a module (to be precise, a subclass
of ModuleType). The Namespace metaclass returns a new module object,
containing all the names I defined inside the class, after modifying any
function objects to treat the new module object as their globals.


Unresolved questions
=

Would a decorator be better than a metaclass?

@namespace
class Example:
...


How about a mechanism to exclude functions from having their globals
changed?


Some limitations

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Thomas Heller]

Changes by Thomas Heller :


--
nosy: +theller

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Paul Moore]

Paul Moore added the comment:

Thanks for the explanation. Based on what's been said, I'd have no objections 
to this, on a "you don't pay for what you don't use" basis - i.e., users who 
don't enable AMSI should not pay any cost for its existence.

I'd be extremely happy if Python was viewed as a high-quality option in the 
server management space, so if this helps in that goal, then that's good.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

And in case it's not clear, I *totally* recognize that AMSI is not for 
everyone. If you're running in a developer environment, your security is almost 
certainly so lax that it's irrelevant.

However, once you start getting serious about what's deployed on your 
production machines, Python gets ruled out very quickly because it lacks 
features like AMSI. Sysadmins don't want to enable a potential attack tool 
unless it's value drastically outweighs the risks (and on Windows, this is 
generally/currently not true), or there is sufficient insight into the behavior 
and use of the tool.

If we want Python to be seen as a production tool and not just a developer tool 
(which I do), we need to address these concerns.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

> what's to stop the attacker from distributing their own interpreter that just 
> doesn't use AMSI?

AppLocker https://technet.microsoft.com/en-us/library/ee619725.aspx

(In short, restrict which executables can be run on a particular system by 
path/certificate/etc.)

Also a combination of ACLs and the fact that they may not be able to copy files 
onto the system directly anyway - see my post just before yours.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Zachary Ware]

Zachary Ware added the comment:

> But in that case, why hook into exec? The malware author can execute 
> arbitrary Python so doesn't *need* exec.

As I understand it, the malware is distributed in encrypted form (probably 
encrypted differently each time it propagates) so as to be given a green-light 
by anti-malware software, then decrypted and run via exec so that the bad code 
is never actually on disk, and thus never scanned.  Yes, the attacker can run 
arbitrary Python code, but if he just distributed the code in plain text, it 
could be detected and blocked.  The unpacking code is simple and generic enough 
that it can't be blocked.


As far as actually enabling AMSI, I'm +0.  I don't understand it well enough to 
be +1, and I share Paul's concerns about startup overhead.  I'm also unsure 
that AMSI actually affords any protection: what's to stop the attacker from 
distributing their own interpreter that just doesn't use AMSI?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

> So the malicious payload is the whole python command, not just file.bin

Yeah, sorry that wasn't clear. Many vulnerabilities allow attackers to schedule 
process launches (e.g. via cron/Task Scheduler/etc.) without actually being 
able to add any files to the machine - Stuxnet took advantage of this, for 
example. So if Python is already there, you can schedule "python -c "import 
urllib, base64; exec(...)"" to download->decode->exec arbitrary code without 
touching the file system or network with obvious sources.

(Right now, I understand base64 is sufficient encryption, at least until the 
antimalware companies add signatures for base64-encoded scripts. Even then, the 
slightest customization of the original code is going to break base64 enough to 
avoid detection, whereas the signatures are flexible enough to handle 
variations to source code.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Paul Moore]

Paul Moore added the comment:

>> I am puzzled as to why "use safe_exec rather than exec" isn't an option

> Because you're going to have a hard time convincing malware authors to use it.

:-) So the malicious payload is the whole python command, not just file.bin. 
OK, fair enough. But in that case, why hook into exec? The malware author can 
execute arbitrary Python so doesn't *need* exec.

As I say, though, I'm not an expert in security threats, so I'm OK with 
accepting that there's a hole here and the proposal plugs it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Zachary Ware]

Zachary Ware added the comment:

> I am puzzled as to why "use safe_exec rather than exec" isn't an option

Because you're going to have a hard time convincing malware authors to use it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Paul Moore]

Paul Moore added the comment:

OK, so a 3rd party module providing a "safe_exec" function would make a good 
proof of concept, I assume. You could probably do that using comtypes or 
pywin32.

I'm not going to try to say what is or isn't a security threat, that's not my 
expertise. But I am puzzled as to why "use safe_exec rather than exec" isn't an 
option, but "use python with the malware scanning option enabled" is. Maybe 
it's like the Powershell execution policy model, though.

I still don't want it to scan my trusted scripts, though. More interpreter 
startup overhead? No thanks.

Anyway, thanks for the clarification. It's early days yet to be debating this 
level of detail, so I'll leave it there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

AMSI is intended for local scanners that are entirely on your own machine, so 
code never goes anywhere, and everything that passes through the file system is 
already scanned because of hooks whether you wrote it or not (maybe you're 
thinking of SmartScreen?).

What this would add is scanning at the exec point in:

python -c exec(decrypt(open('file.bin', 'rb')))

Currently, malware scanners don't get to see the decrypted code, and I'm 
assured this is a common pattern for getting malware onto machines (yes, in 
Python).

That said, I fully expect the official releases to require a registry key to 
enable it (can't be env or CLI option or an attacker would just leave it out :) 
). Wouldn't be on for normal use, but would be available for paranoid people.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Paul Moore]

Paul Moore added the comment:

Strong -1 on anything that scans my locally-written scripts by default. There's 
no reason or justification for that.

Maybe there's a point in having a way to submit an untrusted Python code 
snippet for scanning, but why would that need to be a core service, as opposed 
to a 3rd party module?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26137] [idea] use the Microsoft Antimalware Scan Interface

[Steve Dower]

Steve Dower added the comment:

This now depends on issue27417, since we can't enable AMSI without enabling 
COM, and doing that has a number of back-compat implications.

--
dependencies: +Call CoInitializeEx on startup

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Do you think a DB based on Coroutine and AsyncIO is a good idea? I have written a demo on GitHub.

[Lawrence D’Oliveiro]

On Thursday, May 26, 2016 at 8:56:44 AM UTC+12, jimz...@gmail.com wrote:
> I think in-process DB is quite popular in less serious development, e.g. 
> SQLite.

“less serious”!? It’s the world’s most popular DBMS! There’s likely a copy in 
your pocket or purse right now.
-- 
https://mail.python.org/mailman/listinfo/python-list