[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment:
Not if one of the attributes is something that cannot be part of a typename:
>>> fields = ['def', '-']
>>> namedtuple ('test', fields, rename=True).__doc__
'test(_0, _1)'
>>> namedtuple ('__'.join (fields), fields, rename=True).__doc__
Traceback (most recent call last):
File "", line 1, in
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/collections.py",
line 339, in namedtuple
'alphanumeric characters and underscores: %r' % name)
ValueError: Type names and field names can only contain alphanumeric
characters and underscores: 'def_-'
>>>
Which I admit is a weird thing to be doing, but duplicating attribute names
or trying to use a keyword as an attribute name (or anything else that
requires rename=True) is also weird.
Also it's far from clear that the pre-renaming field names are what is
wanted in the auto-generated typename. If I was actually using attribute
names that required renaming I would want the auto-generated typename to
match the renamed attributes. The original fieldnames play no part in the
operation of the namedtuple class or its instances once it has been
created: only the renamed fieldnames even remain reachable from the
namedtuple object.
Anyway I think I'm probably out at this point. I think Python development
is not a good cultural fit for me, based on this discussion. Which is
weird, since I love working in Python. I even like the whitespace
indentation, although admittedly not quite as much as I thought I would
before I tried it. I hugely enjoy the expressiveness of the language
features, combined with the small but useful set of immediately-available
library functions, together with the multitude of importable standard
modules backing it all up. But I should have known when functools.compose
(which ought to be almost the first thing in any sort of "functional
programming" library) was rejected that I should stay away from attempting
to get involved in the enhancement side of things.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31085>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment:
OK, so it's pretty clear this is heading towards a rejection, but I can't
help but respond to your points:
On 2 August 2017 at 01:12, Raymond Hettinger <rep...@bugs.python.org> wrote:
* This would be a potentially confusing addition to the API.
>
I'm giving a natural meaning to providing a None where it is not permitted
now. The meaning is to provide a reasonable value for the missing
parameter. How could that be confusing? Also it's completely ignorable -
people don't have to pass None and get the auto-generated typename if they
don't want to.
> * It may also encourage bad practices that we don't want to see in real
> code.
>
What bad practices? There are lots of times when providing an explicit name
is a waste of effort. This provides a simple way of telling the library to
figure it out. Aren't there supposedly just two hard things in computer
science? Naming things, and cache invalidation. An opportunity to avoid
naming things that don't need to be specifically named is something worth
taking.
> * We want to be able to search for the namedtuple definition, want to have
> a meaningful repr, and want pickling to be easy.
>
You mean by searching for the typename in the source code? In my primary
usecase, the typename is computed regardless, so it doesn't appear in the
source code and can't be searched for. The other suggestion which appeared
at one point was passing "_" as the typename. This is going to be somewhat
challenging to search for also.
As to the meaningful repr, that is why I want auto-generation of the
typename. This is not for uses like this:
MyType = namedtuple ('MyType', ['a', 'b', 'c'])
It is for ones more like this:
rowtype = namedtuple (None, row_headings)
Or as it currently has to be:
rowtype = namedtuple ('rowtype', row_headings)
(leading to all the rowtypes being the same name, so less meaningful)
Or:
rowtype = namedtuple ('__'.join (row_headings), row_headings)
(which repeats the irrelevant-in-its-details computation wherever it is
needed and doesn't support rename=True, unless a more complicated
computation that duplicates code inside of namedtuple() is repeated)
Finally I'm not clear on how pickling is made more difficult by having
namedtuple() generate a typename. The created type still has a typename.
But I'm interested - this is the only point I don't think I understand.
* This doesn't have to be shoe-horned into the namedtuple API. If an
> actual need did arise, it is trivial to write a wrapper that specifies
> whatever auto-naming logic happens to make sense for a particular
> application:
>
> >>> from collections import namedtuple
> >>> def auto_namedtuple(*attrnames, **kwargs):
> typename = '_'.join(attrnames)
> return namedtuple(typename, attrnames, **kwargs)
>
> >>> NT = auto_namedtuple('name', 'rank', 'serial')
> >>> print(NT.__doc__)
> name_rank_serial(name, rank, serial)
Your code will not work if rename=True is needed. I don't want to repeat
the rename logic as doing so is a code smell.
In short, I'm disappointed. I'm not surprised to make a suggestion, and
have people point out problems. For example, my original proposal ignored
the difficulties of creating the C implementation, and the issue of
circular imports, and I very much appreciated those criticisms. But I am
disappointed at the quality of the objections to these modified proposals.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31085>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment: On 1 August 2017 at 14:32, R. David Murray <rep...@bugs.python.org> wrote: > > R. David Murray added the comment: > > I wrote a "parameterized tests" extension for unittest, and it has the > option of autogenerating the test name from the parameter names and > values. I've never used that feature, and I am considering ripping it out > before I release the package, to simplify the code. If I do I might > replace it with a hook for generating the test name so that the user can > choose their own auto-naming scheme. > > Perhaps that would be an option here: a hook for generating the name, that > would be called where you want your None processing to be? That would not > be simpler than your proposal, but it would be more general (satisfy more > use cases) and might be worth the cost. On the other hand, other > developers might not like the API bloat ;) > It's August, not April. Raymond Hettinger is accusing my proposed API of being potentially confusing, while you're suggesting providing a hook? All I want is the option of telling namedtuple() to make up its own typename, for situations where there should be one but I don't want to provide it. Having said that, if people really think a hook like this is worth doing, I'll implement it. But I agree that it seems excessively complicated. Let's see if auto-generation is useful first, then if somebody wants a different auto-generation, provide the capability. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31085> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment: First, another note I would like to point out: this is much nicer to write within namedtuple than as a wrapper function because it is trivial to use the existing rename logic when needed, as seen in the diff I provided. I suppose I could write a wrapper which calls namedtuple and then changes the class name after creation but that just feels icky. The only other alternatives would be to duplicate the rename logic or have the wrapper not work with rename. By way of response to R. David Murray: Every use case, of everything, is specialized. Another way of thinking of what I'm suggesting is that I would like to make providing a typename optional, and have the library do its best based on the other information provided in the call to namedtuple. This pretty well has to mean mashing the fieldnames together in some way because no other information about the contents of the namedtuple is provided. So I think this is a very natural feature: what else could it possibly mean to pass None for the typename? If for a particular application some other more meaningful auto-generated name is needed, that could still be provided to namedtuple(). For example, an ORM that uses the underlying table name. In response to other suggestions, I don't see how one can prefer "_" all over the place in debugging output to a string that identifies the fieldnames involved. Or really, just the option of having a string that identifies the fieldnames: I'm not forcing anyone to stop passing '_'. To INADA Naoki: thanks for pointing that out. I agree that in the subclass case it no longer matters what typename is used for the namedtuple itself. But isn't that a good reason to allow skipping the parameter, or (since you can't just skip positional parameters) passing an explicit None? On 1 August 2017 at 11:02, R. David Murray <rep...@bugs.python.org> wrote: > > R. David Murray added the comment: > > I think the "vaguely" pretty much says it, and you are the at least the > first person who has *requested* it :) > > This is one of those cost-versus-benefit calculations. It is a > specialized use case, and in other specialized use cases the "automatically > generated" name that makes the most sense is likely to be something > different than an amalgamation of the field names. > > So I vote -0.5. I don't think even the small complication of the existing > code is worth it, but I'm not strongly opposed. > > -- > nosy: +r.david.murray > > ___ > Python tracker <rep...@bugs.python.org> > <http://bugs.python.org/issue31085> > ___ > -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31085> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31086] Add namedattrgetter function which acts like attrgetter but uses namedtuple
Isaac Morland added the comment:
Maybe the issue is that I work with SQL constantly. In SQL, if I say "SELECT
a, b, c FROM t" and table t has columns a, b, c, d, e, f, I can still select a,
b, and c from the result. So to me it is natural that getting a bunch of
attributes returns something (row or object, depending on the context), where
the attributes are still labelled.
I understand why this was rejected as a universal change to attrgetter - in
particular, I didn't re-evaluate the appropriateness of the change once I
realized that attrgetter has a C implementation - but I don't understand why
this isn't considered a natural option to provide.
Using rename=True is just a way of having it not blow up if an attribute name
requiring renaming is supplied. I agree that actually using such an attribute
requires either guessing the name generated by the rename logic in namedtuple
or using numeric indexing. If namedtuple didn't have rename=True then I
wouldn't try to re-implement it but since it does I figure it's worth typing ",
rename=True" once - it's hardly going to hurt anything.
Finally as to use cases, I agree that if the only thing one is doing is sorting
it doesn't matter. But with groupby it can be very useful. Say I have an
iterator providing objects with fields (heading_id, heading_text, item_id,
item_text). I want to display each heading, followed by its items.
So, I groupby attrgetter ('heading_id', 'heading_text'), and write a loop
something like this:
for heading, items in groupby (source, attrgetter ('heading_id',
'heading_text')):
# display heading
# refer to heading.heading_id and heading.heading_text
for item in items:
# display item
# refer to item.item_id and item.item_text
Except I can't, because heading doesn't have attribute names. If I replace
attrgetter with namedattrgetter then I'm fine. How would you write this? In
the past I've used items[0] but that is (a) ugly and (b) requires "items =
list(items)" which is just noise.
I feel like depending on what is being done with map and filter you could have
a similar situation where you want to refer to the specific fields of the tuple
coming back from the function returned by attrgetter.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31086>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment:
I want a meaningful name to appear in debugging output generated by repr() or
str(), not just _ all over the place. I just don't want to specifically come
up with the meaningful name myself.
Right now I pass in the same generated name ('__'.join (field_names)) to the
constructor, but this means I need to repeat that logic in any other similar
application, and I would have to put in special handling if any of my attribute
names required renaming.
I would rather be explicit that I'm not providing a specific name. With your
'_' suggestion it looks like a magic value - why '_'? By specifying None, it's
obvious at the call point that I'm explicitly declining to provide a name, and
then the code generates a semi-meaningful name automatically.
Also, please note that I moved the place where typename is assigned to after
the part where it handles the rename stuff, so the generated names
automatically incorporate a suitable default and remain valid identifiers.
I'm having trouble seeing the downside here. I'm adding one "is None" check
and one line of code to the existing procedure. I can't believe I'm the only
person who has wanted to skip making up a type name but still wanted something
vaguely meaningful in debug output.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31085>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31086] Add namedattrgetter function which acts like attrgetter but uses namedtuple
Isaac Morland added the comment: Here is the diff. Note that I assume implementation of #31085, which allows me to push determination of a name for the namedtuple down into namedtuple itself: diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 62cf708..d507d23 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -14,7 +14,8 @@ list, set, and tuple. ''' -__all__ = ['deque', 'defaultdict', 'namedtuple', 'UserDict', 'UserList', +__all__ = ['deque', 'defaultdict', 'namedtuple', 'namedattrgetter', +'UserDict', 'UserList', 'UserString', 'Counter', 'OrderedDict', 'ChainMap'] # For backwards compatibility, continue to make the collections ABCs @@ -23,7 +24,7 @@ from _collections_abc import * import _collections_abc __all__ += _collections_abc.__all__ -from operator import itemgetter as _itemgetter, eq as _eq +from operator import itemgetter as _itemgetter, attrgetter as _attrgetter, eq as _eq from keyword import iskeyword as _iskeyword import sys as _sys import heapq as _heapq @@ -451,6 +452,14 @@ def namedtuple(typename, field_names, *, verbose=False, rename=False, module=Non return result +def namedattrgetter (attr, *attrs): +ag = _attrgetter (attr, *attrs) + +if attrs: +nt = namedtuple (None, (attr,) + attrs, rename=True) +return lambda obj: nt._make (ag (obj)) +else: +return ag ### Counter -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31086> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31086] Add namedattrgetter function which acts like attrgetter but uses namedtuple
New submission from Isaac Morland: This is meant to replace my proposal in #30020 to change attrgetter to use namedtuple. By creating a new function implemented in Python, I avoid making changes to the existing attrgetter, which means that both the need of implementing a C version and the risk of changing the performance or other characteristics of the existing function are eliminated. My suggestion is to put this in the collections module next to namedtuple. This eliminates the circular import problem and is a natural fit as it is an application of namedtuple. -- components: Library (Lib) messages: 299534 nosy: Isaac Morland priority: normal severity: normal status: open title: Add namedattrgetter function which acts like attrgetter but uses namedtuple type: enhancement versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31086> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
Isaac Morland added the comment:
I'm hoping to make a pull request but while I figure that out here is the diff:
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 8408255..62cf708 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -384,7 +384,6 @@ def namedtuple(typename, field_names, *, verbose=False,
rename=False, module=Non
if isinstance(field_names, str):
field_names = field_names.replace(',', ' ').split()
field_names = list(map(str, field_names))
-typename = str(typename)
if rename:
seen = set()
for index, name in enumerate(field_names):
@@ -394,6 +393,10 @@ def namedtuple(typename, field_names, *, verbose=False,
rename=False, module=Non
or name in seen):
field_names[index] = '_%d' % index
seen.add(name)
+if typename is None:
+typename = '__'.join (field_names)
+else:
+typename = str(typename)
for name in [typename] + field_names:
if type(name) is not str:
raise TypeError('Type names and field names must be strings')
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31085>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31085] Add option for namedtuple to name its result type automatically
New submission from Isaac Morland: I would like to have the possibility of creating a namedtuple type without explicitly giving it a name. I see two major use cases for this: 1) Automatic creation of namedtuples for things like CSV files with headers (see #1818) or SQL results (see #13299). In this case at the point of calling namedtuple I have column headings (or otherwise automatically-determined attribute names), but there probably isn't a specific class name that makes sense to use. 2) Subclassing from a namedtuple invocation; I obviously need to name my subclass, but the name passed to the namedtuple invocation is essentially useless. My idea is to allow giving None for the typename parameter of namedtuple, like this: class MyCustomBehaviourNamedtuple (namedtuple (None, ['a', 'b'])): ... In this case namedtuple will generate a name based on the field names. This should be backward compatible because right now passing None raises a TypeError. So there is no change if a non-None typename is passed, and an exception is replaced by computing a default typename if None is passed. Patch to follow. -- components: Library (Lib) messages: 299532 nosy: Isaac Morland priority: normal severity: normal status: open title: Add option for namedtuple to name its result type automatically type: enhancement versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31085> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30020] Make attrgetter use namedtuple
Isaac Morland added the comment:
What are the "other issues"?
As to the issue you raise here, that's why I use rename=True.
First create a type with an underscore attribute:
>>> t = namedtuple ('t', ['a', '1234'], rename=True)
(just an easy way of creating such a type; used of namedtuple specifically is
admittedly a bit of a red herring)
Now create an object and illustrate its attributes:
>>> tt = t ('c', 'd')
>>> tt.a
'c'
>>> tt._1
'd'
Now use my modified attrgetter to get the attributes as a namedtuple:
>>> attrgetter ('a', '_1') (tt)
attrgetter(a='c', _1='d')
>>>
And the example from the help, used in the test file I've already attached,
illustrates that the dotted attribute case also works.
Essentially, my patch provides no benefit for attrgetter specified attributes
that aren't valid namedtuple attribute names, but because of rename=True it
still works and doesn't break anything. So if you give "a" as an attribute
name, the output will have an "a" attribute; if you give "_b" as an attribute
name, the output will have an "_1" (or whatever number) attribute. Similarly,
it doesn't help with dotted attributes, but it doesn't hurt either.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30020>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30020] Make attrgetter use namedtuple
Isaac Morland added the comment:
I've attached a file which illustrates what I'm proposing to happen with the
examples from the help. Note that attrgetter (attr) is not affected, only
attrgetter (*attrs) for more than one attribute. The idea is that tuples
resulting from attrgetter functions will retain the attribute names from the
original object. In some work I have done recently this would have been very
handy with groupby.
I had some initial confusion because changing the Python attrgetter
implementation didn't make any difference. Once I realized I needed to turn
off import of the C implementation, I figured the rest out fairly quickly.
Here is the diff:
diff --git a/Lib/operator.py b/Lib/operator.py
index 0e2e53e..9b2a8fa 100644
--- a/Lib/operator.py
+++ b/Lib/operator.py
@@ -247,8 +247,12 @@ class attrgetter:
else:
self._attrs = (attr,) + attrs
getters = tuple(map(attrgetter, self._attrs))
+
+from collections import namedtuple
+nt = namedtuple ('attrgetter', self._attrs, rename=True)
+
def func(obj):
-return tuple(getter(obj) for getter in getters)
+return nt._make (getter(obj) for getter in getters)
self._call = func
def __call__(self, obj):
@@ -409,7 +413,7 @@ def ixor(a, b):
try:
-from _operator import *
+pass
except ImportError:
pass
else:
There are some issues that still need to be addressed. The biggest is that
I've turned off the C implementation. I assume that we'll need a C
implementation the new version. In addition to this:
1) I just call the namedtuple type "attrgetter". I'm thinking something
obtained by mashing together the field names or something similar might be more
appropriate. However, I would prefer not to repeat the logic within namedtuple
that deals with field names that aren't identifiers. So I'm wondering if maybe
I should also modify namedtuple to allow None as the type name, in which case
it would use an appropriate default type name based on the field names.
2) I import from collections inside the function. It didn't seem to work at
the top-level, I'm guessing because I'm in the library and collections isn't
ready when operator is initialized. This may be fine I just point it out as
something on which I could use advice.
I'm hoping this provides enough detail for people to understand what I'm
proposing and evaluate whether this is a desireable enhancement. If so, I'll
dig into the C implementation next, although I may need assistance with that.
--
Added file: http://bugs.python.org/file46791/test_attrgetter.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30020>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30020] Make attrgetter use namedtuple
New submission from Isaac Morland: I would find it useful if the tuples returned by attrgetter functions were namedtuples. An initial look at the code for attrgetter suggests that this would be an easy change and should make little difference to performance. Giving a namedtuple where previously a tuple was returned seems unlikely to trigger bugs in existing code so I propose to simply change attrgetter rather than providing a parameter to specify whether or not to use the new behaviour. Patch will be forthcoming but comments appreciated. -- components: Library (Lib) messages: 291314 nosy: Isaac Morland priority: normal severity: normal status: open title: Make attrgetter use namedtuple type: enhancement versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30020> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9299] os.mkdir() and os.makedirs() add a keyword argument to suppress File exists exception.
Isaac Morland ijmor...@uwaterloo.ca added the comment: This exact issue has already been discussed in Issue 1675. -- nosy: +ijmorlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9299 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Isaac Morland ijmor...@uwaterloo.ca added the comment: This is again being discussed in Issue 9299. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1675 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9299] os.mkdir() and os.makedirs() add a keyword argument to suppress File exists exception.
Isaac Morland ijmor...@uwaterloo.ca added the comment: How different is this issue from Issue 1608579, Issue 1239890, Issue 1223238, and Issue 1314067? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9299 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3485] os.path.normcase documentation/behaviour unclear on Mac OS X
New submission from Isaac Morland [EMAIL PROTECTED]:
$ python
Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type help, copyright, credits or license for more information.
from os.path import normcase
normcase ('aB')
'aB'
From http://docs.python.org/lib/module-os.path.html: On Unix, this
returns the path unchanged; on case-insensitive filesystems, it converts
the path to lowercase. Of course, Mac OS X is both Unix and
case-insensitive, which is a rather bizarre combination, but that's it
is. Where is the item for make all file systems case-sensitive, put
the case-insensitivity in the user interface?
--
components: Macintosh
messages: 70571
nosy: ijmorlan
severity: normal
status: open
title: os.path.normcase documentation/behaviour unclear on Mac OS X
type: behavior
versions: Python 2.5
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3485
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3485] os.path.normcase documentation/behaviour unclear on Mac OS X
Isaac Morland [EMAIL PROTECTED] added the comment: Ok, good point. Perhaps the documentation should be updated to clarify that Mac OS is treated as Unix even though a default Mac OS X installation will have a case-insensitive file system. Wouldn't it be possible for a Windows machine to have a case-sensitive file system similarly? Or is it more built-in that the file system on a Windows box will be case-insensitive? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3485 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2269] Problem reporting non-keyword arg after keyword arg syntax error
New submission from Isaac Morland [EMAIL PROTECTED]:
$ cat bug_fine.py
if True:
max (a='a', 'b')
#elif True:
# pass
else:
pass
$ python bug_fine.py
File bug_fine.py, line 2
max (a='a', 'b')
SyntaxError: non-keyword arg after keyword arg
$ cat bug_show.py
if True:
max (a='a', 'b')
elif True:
pass
else:
pass
$ python bug_show.py
Exception exceptions.SyntaxError: ('non-keyword arg after keyword arg',
2) in 'garbage collection' ignored
Fatal Python error: unexpected exception during garbage collection
Abort trap
$
--
components: Interpreter Core
messages: 63448
nosy: ijmorlan
severity: normal
status: open
title: Problem reporting non-keyword arg after keyword arg syntax error
type: compile error
versions: Python 2.5
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2269
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2269] Problem reporting non-keyword arg after keyword arg syntax error
Isaac Morland [EMAIL PROTECTED] added the comment: I should add that the full version information is: Python 2.5 (r25:51908, Aug 15 2007, 11:38:03) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Also, in my actual code (a much larger file, in a project using ll.xist and other libraries), the error manifests differently: I get TypeError: 'int' object is not iterable on startup the first time it loads, then it appears to run subsequent times, but behaves bizarrely. Specifically, *none* of the branches of the if statement run (verified by putting a nonsense symbol after the if statement, and at the beginning and end of each branch of the if statement; the no such symbol error concerns the one *after* the if statement). On neither run is the non-keyword arg after keyword arg reported. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2269 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Isaac Morland added the comment: Attached is an svn diff against the trunk. I was looking at os.py from Python 2.5 not the trunk, and it appears that an attempt at fixing the race condition has already been put into os.py, but I don't believe it's correct. The attached patch renames the existing mkdir to _mkdir, and creates a new mkdir with an additional excl parameter to select error-if-already-exists or not. It defaults to the current behaviour. Similarly, makedirs gets the same extra parameter which is passed down to mkdir. By simply using the new versions as before, one obtains the old behaviour unchanged except that the race condition is corrected. By using excl=False one gets the new behaviour. I have updated the documentation also but I don't really know what I'm doing there so my use of the rst format may not be right. Added file: http://bugs.python.org/file9022/patch.txt __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __Index: Doc/library/os.rst === --- Doc/library/os.rst (revision 59590) +++ Doc/library/os.rst (working copy) @@ -1007,17 +1007,20 @@ .. versionadded:: 2.3 -.. function:: mkdir(path[, mode]) +.. function:: mkdir(path[, mode[, excl=True]]) Create a directory named *path* with numeric mode *mode*. The default *mode* is ``0777`` (octal). On some systems, *mode* is ignored. Where it is used, the current umask value is first masked out. Availability: Macintosh, Unix, Windows. + By default, if the directory already exists, :exc:`OSError` is thrown. If excl is + False, it is not an error for the directory to exist already. + It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. -.. function:: makedirs(path[, mode]) +.. function:: makedirs(path[, mode[, excl=True]]) .. index:: single: directory; creating @@ -1029,6 +1032,10 @@ created. The default *mode* is ``0777`` (octal). On some systems, *mode* is ignored. Where it is used, the current umask value is first masked out. + By default, if the directory already exists, :exc:`OSError` is thrown. If + excl is False, it is not an error for the directory to exist already. It + is never an error for ancestor directories to exist already. + .. note:: :func:`makedirs` will become confused if the path elements to create include Index: Lib/os.py === --- Lib/os.py (revision 59590) +++ Lib/os.py (working copy) @@ -146,29 +146,39 @@ # Super directory utilities. # (Inspired by Eric Raymond; the doc strings are mostly his) +_mkdir = mkdir -def makedirs(name, mode=0777): -makedirs(path [, mode=0777]) +def mkdir(name, mode=0777, excl=True): +mkdir(path [, mode=0777 [, excl=True]]) + +Create the named directory in the indicated mode. By default, if the +directory already exists, OSError is thrown. If excl is False, it is +not an error for the directory to exist already. + +try: +_mkdir(name, mode) +except OSError, e: +if excl or not (e.errno == errno.EEXIST and path.isdir (name)): +raise +def makedirs(name, mode=0777, excl=True): +makedirs(path [, mode=0777 [, excl=True]]) + Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist. This is -recursive. - +recursive. By default, if the directory already exists, OSError is +thrown. If excl is False, it is not an error for the directory to exist +already. It is never an error for ancestor directories to exist already. head, tail = path.split(name) if not tail: head, tail = path.split(head) if head and tail and not path.exists(head): -try: -makedirs(head, mode) -except OSError, e: -# be happy if someone already created the path -if e.errno != errno.EEXIST: -raise +makedirs(head, mode, excl=False) if tail == curdir: # xxx/newdir/. exists if xxx/newdir exists return -mkdir(name, mode) +mkdir(name, mode, excl) def removedirs(name): removedirs(path) ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Isaac Morland added the comment: I should add that the new parameter is called excl by analogy with the O_EXCL option to os.open(). Also, I'm not absolutely certain about the test for which exceptions should be ignored when excl == False: e.errno == errno.EEXIST and path.isdir (name) This will not work if errno is set to something other than EEXIST when mkdir fails due to the directory already existing. The above works on my system but I can't be certain that all mkdir implementations report EEXIST. It should be safe to drop the errno check altogether, and I'm starting to think that we should; at present it's really just an optimization to avoid using .isdir, but only in what should be rather uncommon circumstances. I think the smell of premature optimization may be hard to ignore. So the if statement would be: if excl or not path.isdir (name): raise __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1432] Strange behavior of urlparse.urljoin
Isaac Morland added the comment: RFC 1808 has been obsoleted by RFC 3986: http://tools.ietf.org/html/rfc3986 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1432 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1432] Strange behavior of urlparse.urljoin
Isaac Morland added the comment: Issue 1637, Issue 1779700, and Issue 1462525 also relate to this problem. -- nosy: +ijmorlan __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1432 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
New submission from Isaac Morland: There appears to be a race condition in os.makedirs. Suppose two processes simultaneously try to create two different directories with a common non-existent ancestor. For example process 1 tries to create a/b and process 2 tries to create a/c. They both check that a does not exist, then both invoke makedirs on a. One of these will throw OSError (due to the underlying EEXIST system error), and this exception will be propagated. Note that this happens even though the two processes are trying to create two different directories and so one would not expect either to report a problem with the directory already existing. -- messages: 58919 nosy: ijmorlan severity: minor status: open title: Race condition in os.makedirs type: behavior versions: Python 2.5 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Isaac Morland added the comment: The only thing I found in the bug database concerning os.makedirs was Issue 766910 (http://bugs.python.org/issue766910). I realized os.makedirs had a race condition because in my application I want to create directories but it's perfectly fine if they already exist. This is exactly what trace.py in Issue 766910 seems to need. I started writing my own, which was basically just os.makedirs but calling my own version of os.mkdir which didn't worry about already-existing directories, but realized that wouldn't work. Eventually I ended up with the routines I've put in the attached makedirs.py. I think os.makedirs can be fixed by making what is now its recursive call instead call my version of makedirs. I also think both my mkdir and my makedirs should be present in the standard library as well as the existing versions. Possibly this could be done by adding a flag to the existing versions, defaulted to obtain the current behaviour. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue766910] fix one or two bugs in trace.py
Isaac Morland added the comment: I would suggest that the need to create directories that may already exist (really ensure existence of directories) is not exclusive to trace.py. I am suggesting this be added as an option to os.mkdir and os.makedirs. See Issue 1675. -- nosy: +ijmorlan Tracker [EMAIL PROTECTED] http://bugs.python.org/issue766910 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Changes by Isaac Morland: Added file: http://bugs.python.org/file9016/makedirs.py __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Isaac Morland added the comment: Yes, I'm really combining two things here - the race condition, which I argue is a (minor) bug, and a feature request to be able to ensure exists a directory. I have not produced a proper Python patch before and I have other things to do so this will take longer than one might hope, but I would be happy to create a patch. Note too that the file I uploaded is from my project; I will attempt to make the patch be more appropriate for the standard library than an extract from my project. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1462525] URI parsing library
Isaac Morland added the comment: This is probably overkill, but I've created a Python script (attached) that runs all the tests given in Section 5.4 of RFC 3986. It reports the following: baseurl=http://a/b/c/d;p?q failed for ?y: got http://a/b/c/?y, expected http://a/b/c/d;p?y failed for ../../../g: got http://a/../g, expected http://a/g failed for ../../../../g: got http://a/../../g, expected http://a/g failed for /./g: got http://a/./g, expected http://a/g failed for /../g: got http://a/../g, expected http://a/g failed for http:g: got http://a/b/c/g, expected http:g The last of these is sanctioned by the RFC as acceptable for backward compatibility, so I'll ignore that. The remainder suggest that in addition to the query-relative bug, there is a problem with not reducing /./ to just /, and with dropping excess occurrences of .. that would go above the root. On the other hand, these additional issues are listed in the RFC as abnormal so I'm not sure if people are going to want to put in the time to address them. -- nosy: +ijmorlan Added file: http://bugs.python.org/file8651/testurlparse.py _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1462525 _ testurlparse.py Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
