[issue47174] Define behavior of descriptor-typed fields on dataclasses
Change by mike bayer : -- nosy: +zzzeek ___ Python tracker <https://bugs.python.org/issue47174> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44304] segmentation fault appeared in python 3.10.0b2
mike bayer added the comment: great news! Based on how many random factors were needed to reproduce as well as that it seemed to be gc related and appeared very suddenly, I had an intuition this was on the cpython side, thanks so much for doing this Pablo! -- ___ Python tracker <https://bugs.python.org/issue44304> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44304] segmentation fault appeared in python 3.10.0b2
mike bayer added the comment: yes, if I have time I will begin to undertake that, wanted to put it up here in case anyone has git bisect on speed dial for cpython. -- ___ Python tracker <https://bugs.python.org/issue44304> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44304] segmentation fault appeared in python 3.10.0b2
mike bayer added the comment: if the issue is in greenlet this can be bounced back to https://github.com/python-greenlet/greenlet/issues/242 -- ___ Python tracker <https://bugs.python.org/issue44304> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44304] segmentation fault appeared in python 3.10.0b2
New submission from mike bayer :
segmentation fault related to object deallocation and traceback objects, is
extremely difficult to reproduce and definitely appeared as of 3.10.0b2, does
not occur in 3.10.0b1. linux and osx platforms are affected.
The issue requires "greenlet==1.1.0" to be installed, as well as that for me to
reproduce it I have to use sqlite3 with some special APIs also, so while this
issue might be in greenlet, or sqlite3, I have a feeling these are all factors
that are combining together in a very random way to reveal something that's
actually happening in the interpreter, but in any case would be great if core
devs can see why it happens. The below script has been tested on various linux
platforms, and the overall bug was revealed by an interaction in the SQLAlchemy
test suite that's occurring in many places including OSX, linux. As noted, a
whole bunch of very random things are needed for me to reproduce it.
import greenlet
import sqlite3
class _ErrorContainer(object):
error = None
def _expect_raises_fn(fn):
ec = _ErrorContainer()
try:
fn()
except Exception as err:
assert str(err) == "this is a test"
# assign the exception context outside of the except
# is necessary
ec.error = err
# don't del the exception context is necessary
#del ec
def greenlet_spawn(fn, *args, **kwargs):
# spawning a greenlet is necessary
context = greenlet.greenlet(fn, greenlet.getcurrent())
# assignment to "result" is necessary
result = context.switch(*args, **kwargs)
# raising exception is necessary
raise Exception("this is a test")
class OuterConnectionWrapper:
def __init__(self, connection):
self.connection = connection
def go(self, stmt):
sqlite_connection = self.connection
cursor = sqlite_connection.cursor()
cursor.execute("select 1")
return cursor
def execute(self, stmt):
return greenlet_spawn(self.go, stmt)
def _do_close(self):
self.connection.close()
self.connection = None
def close(self):
self._do_close()
class InnerConnectionWrapper:
def __init__(self, connection):
self.connection = connection
def create_function(self, *arg, **kw):
self.connection.create_function(*arg, **kw)
def cursor(self):
return self.connection.cursor()
def close(self):
self.connection = None
class ConnectionPool:
def __init__(self):
self.conn = sqlite3.connect(":memory:")
def regexp(a, b):
return None
self.conn.create_function("regexp", 2, regexp)
def connect(self):
return InnerConnectionWrapper(self.conn)
def do_test():
pool = ConnectionPool()
def go():
c1 = pool.connect()
conn = OuterConnectionWrapper(c1)
try:
conn.execute("test")
finally:
conn.close()
_expect_raises_fn(go)
do_test()
--
components: Interpreter Core
messages: 395028
nosy: zzzeek
priority: normal
severity: normal
status: open
title: segmentation fault appeared in python 3.10.0b2
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue44304>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43484] valid datetimes can become invalid if the timezone is changed
mike bayer added the comment: > I don't really know why it would be a "security vulnerability", but presumably a library could either convert their datetimes to UTC as soon as they get them from the user if they want to use them as UTC in the future, or they could simply refuse to accept any datetimes outside the range `datetime.datetime.MINYEAR + timedelta(hours=48) < dt.replace(tzinfo=None) < datetime.datetime.MAXYEAR - timedelta(hours=48)`, this is absolutely correct, but I'm not sure if you're aware, there's kind of a whole subsection of the tech community that considers anything that a user might do without understanding all the consequences which could in any way allow untrusted input to affect things to be a "security risk". In SQLAlchemy i had CVEs posted because we have a method called order_by() that accepted a string, and the notion was, someone will write a web app that takes an arbitrary string as input and send it there! CVE! For you and me that would of course be crazy as this is obviously part of the SQL string being sent to the database, but this is a particular programming subculture that has the ability to create a lot of havoc by filling up the CVE system with "Security Vulnerabilities" based on what many of us consider obviously wrong. > Can you clarify why this crashes? Is it because it always returns the > datetime value in UTC? it returns the datetime value in the default timezone set up for the server which could be UTC or a local timezone, but the idea is it's potentially different from the timezone that's been put in. > I'll note that I don't actually understand the difference between > "abstraction layer" and "psycopg2 driver", so I may be conflating those two, from my POV I have always thought PostgreSQLs' TIMESTAMP WITH TIMEZONE datatype is nuts, and that you should only be sending UTC timestamps to a database. But people want to use PG's type and IMO they need to understand what they're doing. thanks for the response. -- ___ Python tracker <https://bugs.python.org/issue43484> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43484] we can create valid datetime objects that become invalid if the timezone is changed
New submission from mike bayer :
So I'm pretty sure this is "not a bug" but it's a bit of a problem and I have a
user suggesting the "security vulnerability" bell on this one, and to be honest
I don't even know what any library would do to "prevent" this.
Basically, the datetime() object limits based on a numerical year of MINYEAR,
rather than limiting based on an actual logical date.
So I can create an "impossible" date as follows:
d = datetime.strptime("Mon Jan 1 00:00:00 0001 +01:00", "%c %z")
or like this:
d = datetime(year=1, month=1, day=1, tzinfo=timezone(timedelta(hours=1)))
andyou can see where this is going - it can't be converted to a timezone
that pushes the year to zero:
>>> from datetime import datetime, timezone, timedelta
>>> d = datetime(year=1, month=1, day=1, tzinfo=timezone(timedelta(hours=1)))
>>> d.astimezone(timezone.utc)
Traceback (most recent call last):
File "", line 1, in
OverflowError: date value out of range
this because, after all, astimezone() is just subraction or addition and if it
overflows past the artificial boundary, well you're out of luck.
Why's this a security problem? ish?because PostgreSQL has a data type
"TIMESTAMP WITH TIMEZONE" and if you take said date and INSERT it into your
database, then SELECT it back using any Python DBAPI that returns datetime()
objects like psycopg2, if your server is in a timezone with zero or negative
offset compared to the given date, you get an error. So the mischievous user
can create that datetime for some reason and now they've broken your website
which can't SELECT that table anymore without crashing.
So, suppose you maintain the database library that helps people send data in
and out of psycopg2.We have, the end user's application, we have the
database abstraction library, we have the psycopg2 driver, we have Python's
datetime() object with MIN_YEAR, and finally we have PostgreSQL with the
TIMEZONE WITH TIMESTAMP datatype that I've never liked.
Of those five roles, whose bug is this?I'd like to say it's the end user
for letting untrusted input that can put unusual timezones and timestamps in
their system. But at the same time it's a little weird Python is letting me
create this date that lacks the ability to convert into UTC. thanks for
reading!
--
components: Library (Lib)
messages: 388585
nosy: zzzeek
priority: normal
severity: normal
status: open
title: we can create valid datetime objects that become invalid if the timezone
is changed
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue43484>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39457] Add an autocommit property to sqlite3.Connection with a PEP 249 compliant manual commit mode and migrate
mike bayer added the comment: I think this issue just discusses the naming of an attribute called ".autocommit". for the discussion for SQLite's unusual starting of transactions, that's all in two other issues: https://bugs.python.org/issue9924 https://bugs.python.org/issue10740 so I would encourage folks to read those discussions. at issue is the limitation of SQLite that it locks the whole file for transactions, which is the main rationale for why SQLite is hesitant to begin a transaction. however, without configurability, this means it's not compatible with SAVEPOINT or serializable isolation levels. when users want to use those two features we have them set isolation_level=None and emit "BEGIN" on the connection directly. the connection.commit() and connection.rollback() methods continue to be functional -- ___ Python tracker <https://bugs.python.org/issue39457> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39457] Add an autocommit property to sqlite3.Connection with a PEP 249 compliant manual commit mode and migrate
mike bayer added the comment:
> Under your proposal, the first line would be changed to say
> "autocommit=True", even though not all the code below is in autocommit mode
> (according to the SQLite engine's definition). What's more, I could insert
> this line of code between statements 3 and 6:
> print("Autocommit mode?", conn.autocommit)
As Marc-Andre indicated, this is in fact how "autocommit" behaves on other
drivers, including:
psycopg2 - uses either connection.autocommit, or
extensions.ISOLATION_LEVEL_AUTOCOMMIT
pymysql - uses conn.autocommit(True|False)
mysqldb - uses conn.autocommit(True|False)
cx_Oracle - uses conn.autocommit
pyodbc - uses conn.autocommit
With all of the above drivers, one can emit "BEGIN" and "COMMIT" using
connection.execute(), and within the scope of that BEGIN/COMMIT pair, the
database "engine" itself is in a transaction. The "driver" however remains in
"autocommit" mode. This mode specifically means the driver is not getting
involved in starting and stopping transactions.
As Marc mentions, we're not supposed to be emitting "BEGIN" and "COMMIT" on
the driver, but none of them get in the way of us doing so, and additionally
most databases support additional options for the "BEGIN/START TRANSACTION"
phase
that aren't available in the DBAPIs so sometimes we don't have much choice at
least for the "BEGIN" command.
Here's an example using psycopg2, where the timestamp now() will freeze
when we're in a transaction started using manual "BEGIN"/ "COMMIT", while
.autocommit stays True, and otherwise match statement execution time if we're
not:
>>> import psycopg2
>>> conn = psycopg2.connect(user="scott", password="tiger",
host="localhost", database="test")
>>> conn.autocommit = True
>>> cursor = conn.cursor()
>>> cursor.execute("SELECT 1")
>>> cursor.execute("select now() = statement_timestamp()")
>>> cursor.fetchall()
[(True,)]
>>> cursor.execute("BEGIN")
>>> cursor.execute("select now() = statement_timestamp();")
>>> cursor.fetchall()
[(False,)] # we're in a transaction
>>> conn.autocommit # still in driver-level autocommit
True
>>> cursor.execute("COMMIT")
>>> cursor.execute("select now() = statement_timestamp();")
>>> cursor.fetchall()
[(True,)]
For SQLAlchemy we already support pysqlite's "isolation_level=None" to
implement "autocommit" so this issue does not affect us much, but the meaning
of the term "autocommit" at the driver level shouldn't be controversial at this
point as there's a lot of precedent. "connection.autocommit" does not refer
to the current transactional state of the database, just the current preference
set upon the driver itself.
--
___
Python tracker
<https://bugs.python.org/issue39457>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16535] json encoder unable to handle decimal
Change by mike bayer : -- nosy: +zzzeek ___ Python tracker <https://bugs.python.org/issue16535> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41629] __class__ not set defining 'X' as . Was __classcell__ propagated to type.__new__?
New submission from mike bayer : This is likely related or a dupe of https://bugs.python.org/issue29270, but the error message is different. I'm posting this to confirm it's the same issue, or not, and to at least provide a google result for people who also see this error as 29270 seems to imply this might not be fixable. Like 29270, it involves the fact that the interpreter seems to be looking at my super() call inside of a method without actually calling it, and then getting upset about __classcell__: from typing import NamedTuple class X(NamedTuple): a: str b: str # comment this out to remove the issue def foo(self): return super(X, self) and that's it! on my interpreter: Python 3.8.3 (default, May 23 2020, 16:34:37) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux I get: $ python test3.py Traceback (most recent call last): File "test3.py", line 4, in class X(NamedTuple): RuntimeError: __class__ not set defining 'X' as . Was __classcell__ propagated to type.__new__? The most surprising thing is that this seems extremely basic and google is not finding this error message for me. -- components: Interpreter Core messages: 375863 nosy: zzzeek priority: normal severity: normal status: open title: __class__ not set defining 'X' as . Was __classcell__ propagated to type.__new__? versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue41629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: I tested "cancellation", shutting down the DB connection mid query. Because the greenlet is only in the middle and not at the endpoints, it propagates the exception and there does not seem to be anything different except for the greenlet sequence in the middle, which is also clear: https://gist.github.com/zzzeek/9e0d78eff14b3bbd5cf12fed8b02bce6 the first comment on the gist has the stack trace produced. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: as far as cancellation, I gather you're referring to what in gevent / greenlet is the GreenletExit exception. Sure, that thing is a PITA. Hence we're all working to provide asyncio frontends and networking backends so that the effects of cancellation I (handwavy handwavy) believe would work smoothly as long as the middle part is done right. cancellation is likely a more prominent issue with HTTP requests and responses because users are hitting their browser stop buttons all the time. With databases this typically is within the realm of network partitioning or service restarts, or if the driver is screwing up in some way which with the monkeypatching thing is more likely, but "cancellation" from a database perspective is not the constant event that I think it would be in an HTTP perspective. > I think I either disagree or am missing something :-). Certainly for both > edgedb and urllib3, when they're running in sync mode, they end up using > synchronous network APIs at the "bottom", and it works fine. OK it took me a minute to understand what you're saying, which is, if we are doing the coroutine.send() thing you illustrated below, we're not in an event loop anyway so we can just call blocking code. OK I did not understand that. I haven't looked at the coroutine internals through all of this (which is part of my original assertion that I should not have been the person proposing this whole greenlet thing anyway :) ). Why did urllib3 write unasync? https://pypi.org/project/unasync/strictly so they can have a python 2 codebase and that's it? SQLAlchemy goes python 3 only in version 2.0. I did bench the coro example against a non-coro example and it's 3x slower likely due to the StopIteration but as mentioned earlier if this is only once per front-to-back then it would not amount to anything in context. Still, the risk factor of a rewrite like that, where risk encompasses just all the dumb mistakes and bugs that would be introduced by rewriting everything, does not seem worth it. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: slight correction: it is of course possible to use gevent with a database driver without monkeypatching, as I wrote my own gevent benchmarks using psycogreen. I think what I'm getting at is that it's a good thing if async DBAPIs could target asyncio explicitly rather than having to write different gevent/eventlet specific things, and that tools like SQLAlchemy can allow for greenlet style coding against those DBAPIs without one having to install/run the whole gevent event loop. Basically I like the greenlet style of coding but I would be excited to skip the gevent part, never do any monkeypatching again, and also have other parts of the app doing asyncio work with other kinds of services. this is about interoperability. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: > Oh, I thought the primary problem for SQLAlchemy supporting async is that the > ORM needs to do IO from inside __getattr__ methods. So I assumed that the > reason you were so excited about greenlets was that it would let you use > await_() from inside those __getattr__ calls, which would involve exposing > your use of greenlets as part of your public API. The primary problem is people want to execute() a SQL statement using await, and then they want to use a non-blocking database driver (basically asyncpg, I'm not sure there are any others, maybe there's one for MySQL also) on the back.Tools like aiopg have provided partial SQLAlchemy-like front-ends to accomplish this but they can't do ORM support, not because the ORM has lazy loading, but just to do explicit operations like query.all() or session.flush() that can sometimes require a lot of front-to-back database operations to complete which would be very involved to rewrite all that code using async/await. Then there's the secondary problem of ORMs doing lazy loading, which is what you refer towards as "IO inside __getattr__ methods". SQLAlchemy is not actually as dependent on lazy loading as other ORMs as we support a wide range of ways to "eagerly" load data up front. With the SQLAlchemy 2.0-style ORM API that has a clear spot for "await" to occur, they can call "await session.execute(select(SomeObject))" and get a whole traversible graph of things loaded up front.We even have a loader called "raiseload" that is specifically anti-lazy loading, it's a loader that raises an error if you try to access something that wasn't explicitly loaded already. So for a lot of cases we are already there. But then, towards your example of "something.b = x", or more commonly in ORMS a get operation like "something.b" emitting SQL, the extension I'm building will very likely include some kind of feature that they can do this with an explicit call. At the moment with the preliminary code that's in there, this might look like: await greenlet_spawn(getattr, something, "b") not very pretty at the moment but that general idea. But the thing is, greenlet_spawn() can naturally apply to anything. So it remains to be seen both how I would want to present this notion, as well as if people are going to be interested in it or not, but as a totally extra thing beyond the "await session.execute()" API that is the main thing, someone could do something like this: await greenlet_spawn(my_business_orm_method) and then in "my_business_orm_method()", all the blocking style ORM things that async advocates warn against could be happening in there. I'm certainly not going to tell people they have to be doing that, but I dont think I should discourage it either, because if the above business method is written "reasonably" (see next paragraph), there really is no problem introduced by implicit IO. By "written reasonably" I'm referring to the fact that in this whole situation, 90% of everything people are doing here are in the context of HTTP services. The problem of, "something.a now creates state that other tasks might see" is not a real "problem" that is solved by using IO-only explicit context switching. This is because in a CRUD-style application, "something" is not going to be a process-local yet thread-global object that had to be created specifically for the application (there's things like the database connection pool and some registries that the ORM uses, but those problems are taken care of and aren't specific to one particular application). There is certainly going to be global mutable state with the CRUD/HTTP application which is the database itself. Event based programming doesn't save you from concurrency issues here because any number of processes maybe accessing the database at the same time. There are well-established concurrency patterns one uses wit h relational databases, which include first and foremost transaction isolation, but also things like compare-and-swap, "select for update", ensuring MVCC is turned on (SQL Server), table locks, etc. These techniques are independent of the concurrency pattern used within the application, and they are arguably better suited to blocking-style code in any case because on the database side we must emit our commands within a transaction serially in any case. The major convenient point of "async" that we can fire off a bunch of web service requests in parallel does not apply to the CRUD-style business methods within our web service request because we can only do things in our ACID transaction one at a time. The problem of "something.a" emitting IO needs to be made sane against other processes also view
[issue22239] asyncio: nested event loop
mike bayer added the comment: > With greenlets OTOH, it becomes possible for another task to observe > someobj.a == 1 without someobj.b == 2, in case someobj.__setattr__ internally > invoked an await_(). let me try this one more time.Basically if someone wrote this: async def do_thing(): someobj.a =1 await do_io_setattr(someobj, "b", 2) then in the async approach, you can again say, assuming "someobj" is global, that another task can observe "someobj.a == 1" without "someobj.b == 2".I suppose you are making the argument that because there's an "await" keyword there, now everything is OK because the reader of the code knows there's a context switch. Whether or not one buys that, the point of my approach is that SQLAlchemy itself *will* publish async methods. End user code *will not* ever context switch to another task without them explicitly calling using an await. That SQLAlchemy internally is not using this coding style, whether or not that leads to new kinds of bugs, there are new kinds of bugs no matter what kind of code a library uses, I don't think this hurts the user community. The community is hurting *A LOT* right now because asyncio is intentionally non-compatible with the traditional blocking approach that is not only still prevalent it's one that a lot of us think is *easier* to work with. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: > With greenlets OTOH, it becomes possible for another task to observe > someobj.a == 1 without someobj.b == 2, in case someobj.__setattr__ internally > invoked an await_(). Any operation can potentially invoke a context switch. > So debugging greenlets code is roughly as hard as debugging full-on > multithreaded code, and much harder than debugging async/await code. I would invite you to look more closely at my approach. The situation you describe above applies to a library like gevent, where IO means a context switch that can go anywhere. My small recipe never breaks out of the asyncio event loop, and it only context switches within the scope of a single coroutine, not to any arbitrary coroutine. So I don't think the above issue applies. Additionally, we are here talking about *libraries* that are independently developed and tested distinct from end-user code.If there's a bug in SQLAlchemy, the end user isn't the person debugging that. arguments over "is async or sync easier to debug" are IMO pretty subjective and at this point they are not relevant to what sync-based libraries should be doing. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: yes so if you have async/await all internal, are you saying you can make that work for synchronous code *without* running the event loop? that is, some kind of container that just does the right thing? my concern with that would still be performance.When asyncio was based on yield and exception throws, that was a lot of overhead to add to functions and that was what my performance testing some years back showed. w/ async/await I'm sure things have been optimized, but in general when i have function a() -> b() -> c(), I am trying to iron as much Python overhead as I possibly can out of that and I'd be concerned that the machinery to work through async/await would add latency. additionally if it was async/await internally but then i need to access the majority of Python DBAPIs that are sync, I need a thread pool anyway, right? which is also another big barrier to jump over. It seems you were involved with urllib3's approach to use a code rewriter rather than a runtime approach based on the discussion at https://github.com/urllib3/urllib3/issues/1323 , but it's not clear if Python 2 compatibility was the only factor or if the concern of "writing a giant shim" was also. -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: > This recipe was one of the reasons why I added `loop.set_task_factory` > method to the spec, so that it's possible to implement this in an *existing* > event loop like uvloop. But ultimately asyncio is flexible enough to let > users use their own event loop which can be compatible with greenlets among > other improvements. Right, when I sought to look at this, I know that my users want to use the regular event loop in asyncio or whatever system they are using. > Ultimately, asyncio will probably never ship with greenlets integration > enabled by default, but we should definitely make it possible (if there are > some limitations now). It doesn't seem to me that nested event loops are > needed for this, right? So right, the approach I came up with does not need nested event loops and it's my vague understanding that nesting event loops is more difficult to debug, because you have these two separate loops handing off to each other. What is most striking about my recipe is that it never even leaves the default event loop. Originally my first iteration when I was trying to get it working, I had a separate thread going on, as it seemed intuitive that "of course you need a thread to bridge async and sync code" but then I erased the "Thread()" part around it and then it just worked anyway. Like it's simple enough that shipping this as a third party library is almost not even worth it, you can just drop this in wherever. If different libraries had their own drop-in of this, they would even work together. greenlet is really like totally cheating. the philosophical thing here is, usually in my many twitter debates on the subject, folks get into how they like the explicit async and await keywords and they like that IO is explicit. So I'm seeking to keep these people happy and give then "async def execute(sql)", and use an async DB driver, but then the library that these folks are using is internally not actually explicit IO. But they don't have to see that, I've done all the work of the "implicit IO" stuff and in a library it's not the same kind of problem anyway. I think this is a really critical technique to have so that libraries that mediate between a user-facing facade and TCP based backends no longer have to make a hard choice about if they are to support sync vs. async (or async with an optional sync facade around it). -- ___ Python tracker <https://bugs.python.org/issue22239> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22239] asyncio: nested event loop
mike bayer added the comment: hey there, I seem to have two cents to offer so here it is.An obscure issue in the Python bug tracker is probably not the right place for this so consider this as an early draft of something that maybe I'll talk about more elsewhere. > This basically divides code into two islands - async and non-async yes, this is the problem, and at the bottom of this apparently somewhat ranty comment is a solution, and the good news is that it does not require Python or asyncio be modified. My concern is kind of around how it is that everyone has been OK with the current state of affairs for so long, why it is that "asyncio is fundamentally incompatible with library X" is considered to be acceptable, and also how easy it was to find a workaround, this is not something I would have expected to come up with. Kind of like you don't expect to invent Velcro or windshield wipers. asyncio's approach is what those of us in the library/framework community call "explicit async", you have to mark functions that will be doing IO and the points at which IO occurs must also be marked.Long ago it was via callback functions, then asyncio turned it into decorators and yields, and finally pep492 turned it into async/await, and it is very nicely done. It is of course a feature of asyncio that writing out async/await means your code can in theory be clearer as to where IO occurs and all that, and while I don't totally buy that myself, I'm of course in favor of that style of coding being available, it definitely has its own kind of self-satisfaction built in when you do it. That's all great. But as those of us in the library/framework community also know, asyncio's approach essentially means, libraries like Flask, Django, my own SQLAlchemy, etc. are all automatically "non-workable" with the asyncio approach; while these libraries can certainly have asyncio endpoints added to them, the task as designed is not that simple, since to go from an asyncio endpoint all the way through library code that doesn't care about async and then down into a networking library that again has asyncio endpoints, the publishing of "async" and the "await" or yield approach must be wired all the way through every function and method. This is all despite that when you're not at the endpoints, the points at which IO occurs is fully predictable such that libraries like gevent don't need you to write it. So we are told that libraries have to have full end-to-end rewrites of all their code to work this way, or otherwise maintain two codebases, or something like that. The side effect of this is that a whole bunch of library and framework authors now get to create all new libraries and frameworks, which do exactly the same thing as all the existing libraries and frameworks, except they sprinkle the "async/await" keywords throughout middle tiers as required. Vague claims of "framework X is faster because it's async" appear, impossible to confirm as it is unknown how much of their performance gains come from the "async" aspect and how much of it is that they happened to rewrite a new framework from scratch in a completely different way (hint: it's the latter). Or in other cases, as if to make it obvious how much the "async/await" keywords come down to being more or less boilerplate for the "middle" parts of libraries, the urllib3 project wrote the "unasync" project [1] so that they can simply maintain two separate codebases, one that has "async/await" and the other which just search-and-replaced them out. SQLAlchemy has not been "replaced" by this trend as asyncio database libraries have not really taken off in Python, and there are very few actual async drivers. Some folks have written SQLAlchemy-async libraries that use SQLAlchemy's expression system while they have done the tedious, redundant and impossible-to-maintain work of replicating enough of SQLAlchemy's execution internals such that a modest "sqlalchemy-like" experience with asyncio can be reproduced. But these libraries are closed out from all of the fixes and improvements that occur to SQLAlchemy itself, as well as that these systems likely target a smaller subset of SQLAlchemy's behaviors and features in any case.They certainly can't get the ORM working as the ORM runs lots of SQL executions internally, all of which would have to propagate their "asyncness" outwards throughout hundreds of functions. The asyncpg project, one of the few asyncio database drivers that exists, notes in its FAQ "asyncpg uses asynchronous execution model and API, which is fundamentally incompatible with SQLAlchemy" [2], yet we know this is not true because SQLAlchemy works just fine with gevent and eventlet, with no architectural changes at all. Us
[issue38749] sqlite3 driver fails on four byte unicode strings coming from JSON_EXTRACT
mike bayer added the comment: silly me thinking python devs had better access to SQLite devs :) -- ___ Python tracker <https://bugs.python.org/issue38749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38749] sqlite3 driver fails on four byte unicode strings coming from JSON_EXTRACT
mike bayer added the comment: Hi where did you report it?I don't see it on the mailing list or in their fossil tracker. -- ___ Python tracker <https://bugs.python.org/issue38749> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38749] sqlite3 driver fails on four byte unicode strings coming from JSON_EXTRACT
New submission from mike bayer :
When using unicode characters inside of JSON strings, values retrieved via the
JSON_EXTRACT SQLite function fail to be decoded by the sqlite3 driver if they
include four-byte unicode characters.
Version information for my build, which is Fedora 30:
Python 3.7.4 (default, Jul 9 2019, 16:32:37)
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.26.0'
Demo as follows:
import json
import sqlite3
# two unicode strings, the second one has four byte character in it
good_data = "réve illé"
bad_data = "réve illé"
# create simple json structures
good_data_json = json.dumps({"foo": good_data})
bad_data_json = json.dumps({"foo": bad_data})
# all strings are valid utf-8
# data round trips correctly through json
assert json.loads(good_data_json.encode("utf-8").decode("utf-8")) == {
"foo": good_data
}
assert json.loads(bad_data_json.encode("utf-8").decode("utf-8")) == {
"foo": bad_data
}
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
cursor.execute("CREATE TABLE some_data (id INT, data JSON)")
cursor.executemany(
"INSERT INTO some_data(id, data) VALUES(?, ?)",
[(1, good_data_json), (2, bad_data_json)],
)
# we can retrieve the JSON objects as a whole from the DB, no issue
cursor.execute("SELECT some_data.data FROM some_data ORDER BY id")
assert cursor.fetchall() == [(good_data_json, ), (bad_data_json, )]
# when we use JSON_EXTRACT, then full utf-8 support is lost
# extract good value from JSON object
cursor.execute("""
SELECT JSON_EXTRACT(some_data.data, '$."foo"')
FROM some_data WHERE id=1
""")
assert cursor.fetchone()[0] == good_data
# extract bad value from JSON object; utf-8 failure
# sqlite3.OperationalError: Could not decode to UTF-8 column
# 'JSON_EXTRACT(some_data.data, '$."foo"')' with text 'r��ve�� ill��'
cursor.execute("""
SELECT JSON_EXTRACT(some_data.data, '$."foo"')
FROM some_data WHERE id=2
""")
assert cursor.fetchone()[0] == bad_data
output:
Traceback (most recent call last):
File "test4.py", line 50, in
""")
sqlite3.OperationalError: Could not decode to UTF-8 column
'JSON_EXTRACT(some_data.data, '$."foo"')' with text 'r��ve�� ill��'
surprising to say the least as the SQLite driver has always been completely
solid with all unicode, but there you go.
--
components: Library (Lib)
messages: 356257
nosy: zzzeek
priority: normal
severity: normal
status: open
title: sqlite3 driver fails on four byte unicode strings coming from
JSON_EXTRACT
type: crash
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue38749>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37010] Review performance of inspect.getfullargspec
mike bayer added the comment: thanks for creating this issue Nick! -- nosy: +zzzeek ___ Python tracker <https://bugs.python.org/issue37010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36751] Changes in the inspect module for PEP 570
mike bayer added the comment: Just did some benchmarks, and while Signature has apparently had a big speedup in Python 3.6, it is still much less performant than either the Python 2.7 or Python 3.3 implementations, anywhere from 6-18 times slower approximately depending on the function. For those of us that need a getargspec that is only used on selected, internal functions where we don't need the newer language features, it's likely worth it to vendor the Python 3.3 getfullargspec() implementation which is very simple: https://gist.github.com/zzzeek/0eb0636fa3917f36ffd887d9f765c208 -- ___ Python tracker <https://bugs.python.org/issue36751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36751] Changes in the inspect module for PEP 570
mike bayer added the comment: > We are talking again and again that we have a lot of old things in the > standard library but it seems that removing them is also a problem. I agree that the reason we have these deprecation warnings is so that we do get notified and we do fix them. I think Signature is tough here because it is so different from how getfullargspec() worked which makes it difficult to port towards, and its very long and complicated invocation steps, with many loops, function and method calls, object creation overhead, none of which existed in Python 3.3's 20 line getfullargspec() implementation, is making me very hesitant to switch to it and I'm continuing to consider just writing my own thing that keeps the overhead as low as possible; but I will run timing tests on all versions of things before I do anything like that. -- ___ Python tracker <https://bugs.python.org/issue36751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36751] Changes in the inspect module for PEP 570
mike bayer added the comment: > A deprecating warning doesn't hurt: you are still able to run your code. this is very much untrue in modern testing environments, as it is common that test suites fail on deprecation warnings, especially in libraries, to ensure downstream compatibility. My observation is that as Python 3 has been generally a lot more aggressive about new language features than things used to be in the py2k days, a lot more of these deprecations have been coming through. I've looked at pep 570 and this new syntax does not affect the four libraries for which I have to rewrite getfullargspec, so while I'm looking a bit at things like the "Signature" backport for py2.7 (funcsigs), the pure-Python complexity of Signature is very hard to take on as well as a py2k-only dependency so I'm still leaning towards producing the most minimal getfullargspec() I can use that continues to work for "traditional" Python signatures. i understand that Signature is "future-proof", but it is heavy on the pure Python and object creation and does not provide for any simple py2k/py3k cross-compatibility solution. I'd very much prefer that Signature at least be written in C for a few major versions, as well as the community has had time to move further python-3-only, before getfullargspec is so aggressively deprecated. -- ___ Python tracker <https://bugs.python.org/issue36751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36751] Changes in the inspect module for PEP 570
mike bayer added the comment: if a function continues to work correctly throughout the span of a Python X version, why does it need to be removed? I have a foggy memory but I don't seem to recall functions that were technically redundant being aggressively deprecated in the 2.x series. This warning is causing a lot of downstream pain right now and this is after we already all had to deal with getargspec() going away. -- ___ Python tracker <https://bugs.python.org/issue36751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36751] Changes in the inspect module for PEP 570
Change by mike bayer : -- nosy: +zzzeek ___ Python tracker <https://bugs.python.org/issue36751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35398] SQLite incorrect row count for UPDATE
Change by mike bayer : -- nosy: +zzzeek ___ Python tracker <https://bugs.python.org/issue35398> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
mike bayer <mike...@zzzcomputing.com> added the comment: for those watching this would be the findall() case which is consistent between pythons: import re for reg in [ 'VARCHAR(30) COLLATE "en_US"', 'VARCHAR(30)' ]: print(re.findall(r'(?: COLLATE.*)?$', reg)) output (all pythons): [' COLLATE "en_US"', ''] [''] so yes there are two matches for one and only one for the other. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
mike bayer <mike...@zzzcomputing.com> added the comment: for now the quickest solution is to add "count=1" so that it only replaces once. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
mike bayer <mike...@zzzcomputing.com> added the comment: also, removing the "?" is not an option for me. I need the brackets to be placed prior to the "COLLATE" subsection, but unconditionally even if the "COLLATE" section is not present. Looking at the change the behavior seems wrong to me. The regexp states, "match the end of the string, plus an optional "COLLATE" clause, into a capturing expression. replace everything here, e.g. the capturing part as well as the dollar sign part, with a single instance of FOO plus the captured part". It is entirely unintuitive to me how a second replacement would be occurring here. I cannot prove it but I think this change is wrong. I will try to rewrite the expression. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
mike bayer <mike...@zzzcomputing.com> added the comment: can you point me to the documentation? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
mike bayer <mike...@zzzcomputing.com> added the comment: correction, that's fedora 26, not 27 -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32998] regular expression regression in python 3.7
New submission from mike bayer <mike...@zzzcomputing.com>: demo: import re inner = 'VARCHAR(30) COLLATE "en_US"' result = re.sub( r'((?: COLLATE.*)?)$', r'FOO\1', inner ) print(inner) print(result) in all Python versions prior to 3.7: VARCHAR(30) COLLATE "en_US" VARCHAR(30)FOO COLLATE "en_US" in Python 3.7.0b2: VARCHAR(30) COLLATE "en_US" VARCHAR(30)FOO COLLATE "en_US"FOO platform: Fedora 27 python build: Python 3.7.0b2 (default, Mar 5 2018, 09:37:32) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux -- components: Library (Lib) messages: 313251 nosy: zzzeek priority: normal severity: normal status: open title: regular expression regression in python 3.7 type: behavior versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32998> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks
mike bayer added the comment: > Here is a pure Python PoC patch that allows unbounded Queue and LifoQueue to > have reentrant put(). per http://bugs.python.org/msg275377 guido does not want an RLock here. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14976> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks
mike bayer added the comment: yep, that's what im doing in my approach. though longer term thing, I noticed it's very hard to find documentation on exactly when gc might run. E.g. would it ever run if I did something innocuous, like "self.thread_id = None" (probably not). Just wasn't easy to get that answer. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14976> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks
mike bayer added the comment: SQLAlchemy suffered from this issue long ago as we use a Queue for connections, which can be collected via weakref callback and sent back to put(), which we observed can occur via gc.For many years (like since 2007 or so) we've packaged a complete copy of Queue.Queue with an RLock inside of it to work around it. I'm just noticing this issue because I'm writing a new connection pool implementation that tries to deal with this issue a little differently (not using Queue.Queue though). -- nosy: +zzzeek ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14976> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26690] PyUnicode_Decode breaks when Python / sqlite3 is built with sqlite 3.12.0
mike bayer added the comment: i realized this is probably with my build overall. let me do some more testing and ill reopen if i can confirm this more closely. -- resolution: -> works for me status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26690> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26690] PyUnicode_Decode breaks when Python / sqlite3 is built with sqlite 3.12.0
New submission from mike bayer:
So I really don't know *where* the issue is in this one, because I don't know
enough about the different bits.
Step 1: Save this C program to demo.c:
#include
static PyObject *
unicode_thing(PyObject *self, PyObject *value)
{
char *str;
Py_ssize_t len;
if (value == Py_None)
Py_RETURN_NONE;
if (PyString_AsStringAndSize(value, , ))
return NULL;
return PyUnicode_Decode(str, len, "utf-8", "ignore");
}
static PyMethodDef UnicodeMethods[] = {
{"unicode_thing", unicode_thing, METH_O,
"do a unicode thing."},
{NULL, NULL, 0, NULL}/* Sentinel */
};
PyMODINIT_FUNC
initdemo(void)
{
(void) Py_InitModule("demo", UnicodeMethods);
}
Step 2: Build with a setup.py:
from distutils.core import setup, Extension
module1 = Extension('demo',
sources = ['demo.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1])
$ python setup.py build_ext
3. Run with a normal Python that is *not* built against SQLite 3.12.0:
$ python
Python 2.7.10 (default, Sep 8 2015, 17:20:17)
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import demo
>>> demo.unicode_thing('sql')
u'sql'
4. Now build Python 2.7.11 *with* SQLite 3.12.0 in the -I / -L paths. Run the
same program *With* that Python:
$ /opt/Python-2.7.11-sqlite-3.12.0/bin/python
Python 2.7.11 (default, Apr 4 2016, 14:20:47)
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import demo
>>> demo.unicode_thing('sql')
u's\x00q'
Somehow the presence of sqlite-3.12.0 in the build is breaking the Python
interpreter. I think. I really don't know. The bad news is, this is the
code for SQLAlchemy's unicode processor, and as SQlite 3.12.0 starts getting
put in distros, the world is going to break. So this is kind of really hard to
test, I don't understand it, and it's totally urgent. Any insights would be
appreciated!
--
components: Extension Modules
messages: 262864
nosy: zzzeek
priority: normal
severity: normal
status: open
title: PyUnicode_Decode breaks when Python / sqlite3 is built with sqlite 3.12.0
versions: Python 2.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26690>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10740] sqlite3 module breaks transactions and potentially corrupts data
mike bayer added the comment: @rian - your issue for this is http://bugs.python.org/issue9924. The implicit BEGIN in all cases will probably never be the default but we do need an option for this to be the case, in order to support SERIALIZABLE isolation. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10740> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10740] sqlite3 module breaks transactions and potentially corrupts data
mike bayer added the comment: @Rian - implicit transactions are part of the DBAPI spec. Looking at the original description, the purpose of this bug is to not *end* the transaction when DDL is received. So there's no solution for "database is locked" here, other than pysqlite's usual behavior of not emitting BEGIN until DML is encountered (which works well). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10740> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
mike bayer added the comment: thanks for the merge! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
mike bayer added the comment: my star went through. let's merge. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
mike bayer added the comment: hi Yury - I did sign it earlier today. It should have been sent off to the person that manages that, at least that's what the email receipt said. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
mike bayer added the comment: It would be great if the patch could be attached to the issue as a patch file, including some tests. the mantra we all share. I'll take a look. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
mike bayer added the comment: patch w/ test -- keywords: +patch Added file: http://bugs.python.org/file38882/issue23898.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23898] inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__
New submission from mike bayer: this bug appeared in Python 3.4.The inspect.classify_class_attrs compares the identity objects of unknown type using the `==` operator unnecessarily and also evaluates objects of unknown type assuming they return `True` for a straight boolean evaluation. This breaks among other things the ability to use the help() function with SQLAlchemy mapped objects. Demo: class MySpecialObject(object): def __eq__(self, other): return MySpecialObject() def __bool__(self): raise NotImplementedError( This object does not specify a boolean value) class MyClass(object): some_thing = MySpecialObject() import inspect print(inspect.classify_class_attrs(MyClass)) # ultimate goal: help(MyClass) A patch here would be to compare unknown objects for identity using the `is` operator as well as using `is not None` when asserting that an object of unknown type is non-None. This patch resolves: --- inspect_orig.py 2015-04-09 10:28:46.0 -0400 +++ inspect.py 2015-04-09 10:29:37.0 -0400 @@ -380,7 +380,7 @@ # first look in the classes for srch_cls in class_bases: srch_obj = getattr(srch_cls, name, None) -if srch_obj == get_obj: +if srch_obj is get_obj: last_cls = srch_cls # then check the metaclasses for srch_cls in metamro: @@ -388,7 +388,7 @@ srch_obj = srch_cls.__getattr__(cls, name) except AttributeError: continue -if srch_obj == get_obj: +if srch_obj is get_obj: last_cls = srch_cls if last_cls is not None: homecls = last_cls @@ -402,7 +402,7 @@ # unable to locate the attribute anywhere, most likely due to # buggy custom __dir__; discard and move on continue -obj = get_obj or dict_obj +obj = get_obj if get_obj is not None else dict_obj # Classify the object or its descriptor. if isinstance(dict_obj, staticmethod): kind = static method -- components: Library (Lib) messages: 240331 nosy: zzzeek priority: normal severity: normal status: open title: inspect() changes in Python3.4 are not compatible with objects that implement special __bool__, __eq__ versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23898 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22956] Improved support for prepared SQL statements
mike bayer added the comment: prepared statements are, in proportion to the typical speed issues in Python (see my comparison benchmark at https://mail.python.org/pipermail/db-sig/2014-December/006147.html) a fairly small optimization that the DBAPI already allows for in an implicit sense, via an internal statement cache - the execute() method of DBAPI (see https://www.python.org/dev/peps/pep-0249/#id14) allows for this optimization. Therefore an application that wishes to use this optimization with a participating DBAPI only need to maintain a reference to the cursor, and continue to use that same cursor for the same statement - pretty much identically to how it would be used in the explicit prepare step. An explicit prepare step should be no more intrusive than an optional flag sent along to cursor(), as MySQL-connector does, see http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlconnection-cursor.html. The DBAPI does not use objects to represent statements. In JDBC, there is a Statement and PreparedStatement object, but because JDBC has no explicit sense of a cursor, these are in fact just cursor objects (see https://mail.python.org/pipermail/db-sig/2014-December/006168.html for my description of this). Therefore we are really just talking about a potentially modified cursor class, and in Python we can just use a flag, there's no need for heavy-handed Java-esque concepts like new classes. If one really wishes there were a PreparedStatement class, using the flag approach one can have it with very simple code that IMO does not belong in the DBAPI: class PreparedStatement(object): def __init__(self, connection, statement): self.cursor = connection.cursor(prepared=True) self.statement = statement def execute(self, params): self.cursor.execute(self.statement, params) def fetchall(self): return self.cursor.fetchall() # ... etc -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22956 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22956] Improved support for prepared SQL statements
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22956 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14847] AttributeError: NoneType has no attribute 'utf_8_decode'
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14847 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22457] load_tests not invoked in root __init__.py when start=package root
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22457 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21718] sqlite3 cursor.description seems to rely on incomplete statement parsing for detection
New submission from mike bayer: Per DBAPI and pysqlite docs, .description must be available for any SELECT statement regardless of whether or not rows are returned. However, this fails for SELECT statements that aren't simple SELECTs, such as those that use CTEs and therefore start out with WITH:: import sqlite3 conn = sqlite3.connect(:memory:) cursor = conn.cursor() cursor.execute( create table foo (id integer primary key, data varchar(20)) ) cursor.execute( insert into foo (id, data) values (10, 'ten') ) cursor.execute( with bar as (select * from foo) select * from bar where id = 10 ) assert cursor.description is not None cursor.execute( with bar as (select * from foo) select * from bar where id = 11 ) assert cursor.description is not None the second statement returns no rows and cursor.description is None. Libraries like SQLAlchemy which rely on this to determine that the statement supports fetchone() and similar are blocked. -- components: Library (Lib) messages: 220263 nosy: zzzeek priority: normal severity: normal status: open title: sqlite3 cursor.description seems to rely on incomplete statement parsing for detection type: behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21718 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11380] Improve reporting of broken stdout pipe during interpreter shutdown
mike bayer added the comment: my users are reporting one of these issues, and while I can easily catch IOError on Python 2, I can't catch anything on Python 3, and pointing SIGPIPE to SIG_DFL isn't an option because this is for Alembic migrations and the command needs to complete its work. What is/will be the recommended solution for people who want their programs to finish totally but not complain when their pipe is cut off in Python 3? -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11380 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20828] inspect.getargspec() returns wrong answer with datetime.today.__call__()
New submission from mike bayer: this appears like it may be related to http://bugs.python.org/issue20786, at least in terms of inspect.getargspec() seems to be returning answers in 3.4 where it used to raise TypeError, however like in 20786 it's again returning the wrong answer. I'm a little concerned that some change was made to allow inspection of lots of builtins that wasn't before, but the argument lists haven't been checked. test case: import inspect import datetime try: insp = inspect.getargspec(datetime.datetime.today.__call__) except TypeError: pass else: print(insp) assert insp == ([self], args, kwargs, None) # claims to accept args and kwargs, ok let's try... datetime.datetime.today.__call__(1, 2, foo='bar') # raises: # TypeError: today() takes no keyword arguments -- components: Library (Lib) messages: 212556 nosy: zzzeek priority: normal severity: normal status: open title: inspect.getargspec() returns wrong answer with datetime.today.__call__() versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20828 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20786] inspect.getargspec() returns wrong answer with property.__delete__()
mike bayer added the comment: see also http://bugs.python.org/issue20828 as it seems like there might be a bigger pattern here (not sure). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20786 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20828] inspect.getargspec() returns wrong answer with datetime.today.__call__()
mike bayer added the comment: I've also worked around this on my end, so if my poking into today.__call__() isn't really a good idea anyway, I'm not doing that anymore. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20828 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20828] inspect.getargspec() returns wrong answer with datetime.today.__call__()
mike bayer added the comment: we basically need to be able to get the argument signature for anything that passes callable(); function, method, class, object with __call__. So this is the logic we use: import inspect def get_callable_argspec(fn): if inspect.isfunction(fn) or inspect.ismethod(fn): inspectable = fn elif inspect.isclass(fn): inspectable = fn.__init__ elif hasattr(fn, '__call__'): inspectable = fn.__call__ else: inspectable = fn try: return inspect.getargspec(inspectable) except TypeError: raise def callable1(self, x, y): pass class SomeClass(object): def __init__(self, x, y): pass def callable2(self, x, y): pass def __call__(self, x, y): pass callable3 = SomeClass callable4 = SomeClass(2, 3) for callable_ in (callable1, SomeClass(1, 2).callable2, callable3, callable4): assert callable(callable_) # the interpreter can tell me this # how can it reliably tell me this? assert get_callable_argspec(callable_) == ([self, x, y], None, None, None) If you pass a builtin like datetime.datetime.today to it, isfunction()/ismethod()/isclass() return false, but it does have a __call__(). I'm working around this now by just refusing to act on anything that is types.BuiltinMethodType or types.BuiltinFunctionType. Any guidance on what the proper way is to get the argument signature for any object that returns True for callable() would be very helpful (py2.6-3.x compatible). I'm not sure if there's a stdlib call for this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20828 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20828] inspect.getargspec() returns wrong answer with datetime.today.__call__()
mike bayer added the comment: I've got something like that going on right now, but am doing it in the other direction; when I look at __call__(), I only do anything with it if it passes inspect.ismethod(). Since I only want plain Python __call__() functions on plain Python objects. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20828 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20786] inspect.getargspec() returns wrong answer with property.__delete__()
New submission from mike bayer: The Python builtin property() historically does not allow inspect.getargspec to be called on any of __get__(), __set__(), or __delete__(). As of 3.4, it seems that this call now succeeds. However the answer it gives for __delete__() seems to be incorrect. Below illustrates that property.__delete__() accepts two arguments self and instance but inspect is giving a misleading answer: import inspect # userland descriptor class Descriptor(object): def __get__(self, instance, owner): if instance is None: return self def __set__(self, instance, value): pass def __delete__(self, instance): pass # class with property + userland descriptor class X(object): @property def foo(self): pass @foo.deleter def foo(self): pass bar = Descriptor() # property.__delete__ and Descriptor.__delete__ both accept two arguments: property.__delete__(X.foo, X()) Descriptor.__delete__(X.bar, X()) # on all versions, userland __delete__ produces 'self', 'instance' for args assert inspect.getargspec(Descriptor.__delete__) == (['self', 'instance'], None, None, None) try: # but on python 3.4, it returns ['instance'] insp = inspect.getargspec(property.__delete__) assert insp == (['self', 'instance'], None, None, None), insp except TypeError as e: # on all other python versions, raises # slot wrapper '__delete__' of 'property' objects is not a Python function print(Exception: %s % e) -- messages: 212313 nosy: zzzeek priority: normal severity: normal status: open title: inspect.getargspec() returns wrong answer with property.__delete__() versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20786 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20786] inspect.getargspec() returns wrong answer with property.__delete__()
Changes by mike bayer mike...@zzzcomputing.com: -- components: +Library (Lib) type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20786 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20786] inspect.getargspec() returns wrong answer with property.__delete__()
mike bayer added the comment: for context, we are currently creating wrappers around these methods in SQLAlchemy, and in the case of property dunders, we expect that exception and catch it. So when the exception doesn't happen, we assume the answer is correct, but in this case it's not - the answer getargspec() gives us cannot be used to create a correct wrapper unless there's some other detail I'm missing. hence this is backwards incompatible. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20786 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17482] functools.update_wrapper mishandles __wrapped__
mike bayer added the comment: this is actually biting me, I think, though I'm having a very hard time getting a small Python script to duplicate it :/. https://bitbucket.org/zzzeek/alembic/issue/175/test-suite-failure-under-python34 refers to the current problems I'm having. I am not really familiar with the ramifications of __wrapped__ so I need to learn some more about that but also I'm concerned that a standalone test which attempts to simulate everything as much as possible is not doing the same thing. -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17482 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17482] functools.update_wrapper mishandles __wrapped__
mike bayer added the comment: i think I found the problem. sorry for the noise. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17482 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17482] functools.update_wrapper mishandles __wrapped__
mike bayer added the comment: OK well, let me just note what the issue is, and I think this is pretty backwards-incompatible, and additionally I really can't find any reasonable way of working around it except for just deleting __wrapped__. It would be nice if there were some recipe or documentation that could point people to how do do the following pattern: import functools import inspect def my_wrapper(fn): def wrapped(x, y, z): return my_func(x, y) wrapped = functools.update_wrapper(wrapped, fn) return wrapped def my_func(x, y): pass wrapper = my_wrapper(my_func) # passes for 2.6 - 3.3, fails on 3.4 assert inspect.getargspec(wrapper) == (['x', 'y', 'z'], None, None, None), inspect.getargspec(wrapper) basically in Alembic we copy out a bunch of decorated functions out somewhere else using inspect(), and that code relies upon seeing the wrappers list of arguments, not the wrapped. Not that Python 3.4's behavior isn't correct now, but this seems like something that might be somewhat common. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17482 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9924] sqlite3 SELECT does not BEGIN a transaction, but should according to spec
mike bayer added the comment: see also http://bugs.python.org/issue10740, which also relates to pysqlite attempting to make guesses as to when transactions should begin and end. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9924 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10740] sqlite3 module breaks transactions and potentially corrupts data
mike bayer added the comment: well Adam, you might also be surprised to see pysqlite not doing very well on the *other* side of the equation either; that is, when it *begins* the transaction. Issue http://bugs.python.org/issue9924 refers to this and like this one, hasn't seen any activity since 2011. You might be interested in the rationale on that one, which is that python apps may wish to have some degree of concurrency against a sqlite file for an application that is doing all SELECTs. I am certainly in favor of a pure pep-249 approach that emits BEGIN on the first execute() call period, and never implicitly rolls back or commits. However, I disagree this should be enabled by default nor that there should not be any option for the old behavior. I also think the delayed BEGIN feature should still be available to counteract SQLite's particular difficulty with concurrency. If there is code that relies upon a bug in order to function, which would then fail to function in that way if the bug is fixed, then by definition fixing that bug is a backwards-incompatible change. Python std lib can't afford to roll out a change that blatantly backwards-incompatible. The issue regarding comments not being parsed unfortunately should also be a separate issue that needs to be fixed, as nasty as it is that pysqlite tries to parse SQL. I disagree that most users are expecting SQLite's DDL to be transactional; transactional DDL is a luxury feature to many, including the vast numbers of developers that use MySQL, not to mention Oracle, neither of which have any transactional DDL capabilities. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10740 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10740] sqlite3 module should allow DDL statements in transactions
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10740 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9253] argparse: optional subparsers
mike bayer added the comment:
um, this seems like a regression/bug? I now have users complaining that my
apps are broken because of this change as of Python 3.3.My application is
supposed to return the help screen when no command is given. Now I get a
None error because argparse is not trapping this condition:
from argparse import ArgumentParser
parser = ArgumentParser(prog='test')
subparsers = parser.add_subparsers()
subparser = subparsers.add_parser(foo, help=run foo)
parser.parse_args()
$ python3.2 test.py
usage: test [-h] {foo} ...
test: error: too few arguments
$ python3.3 test.py
$
This seems very much like a major feature has been yanked away from argparse,
now I have to check for this condition explicitly.
am I on the right issue here or do I need to open something new ?
--
nosy: +zzzeek
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9253
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16389] re._compiled_typed's lru_cache causes significant degradation of the mako_v2 bench
mike bayer added the comment: in response to ezio, I poked around the source here, since I've never been sure if re.compile() cached its result or not. It seems to be the case in 2.7 and 3.2 also - 2.7 uses a local caching scheme and 3.2 uses functools.lru_cache, yet we don't see as much of a slowdown with 3.2. so it seems like the caching behavior is precedent here, but I would revert re.py's caching scheme to the one used in 2.7 if the functools.lru_cache can't be sped up very significantly. ideally lru_cache would be native. also does python include any kind of benchmarking unit tests ? over in SQLA we have an approach that fails if the call-counts of various functions, as measured by cProfile, fall outside of a known range. it's caught many issues like these for me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16389 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1062277] Pickle breakage with reduction of recursive structures
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1062277 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7511] msvc9compiler.py: ValueError: [u'path']
mike bayer mike...@zzzcomputing.com added the comment: I think it's fair to ask the user to setup the environment correctly before running python setup.py install We're supporting automatic fallback to non-C install for those environments that don't support it. We're just looking for a predictable CCompilerError to be raised for conditions like this rather than a ValueError where we have to parse the string message. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7511 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7511] msvc9compiler.py: ValueError: [u'path']
mike bayer mike...@zzzcomputing.com added the comment: regarding hey this is an MS bug not Python, projects which feature optional C extensions are starting to apply workarounds for the issue on their end (I will need to commit a specific catch for this to SQLAlchemy) - users need to install our software and we need to detect compilation failures as a sign to move on without it. I think it's preferable for Python distutils to work around an MS issue rather than N projects having to work around an MS issue exposed through distutils. Seems like this bug has been out there a real long time...bump ? -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7511 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9924] sqlite3 SELECT does not BEGIN a transaction, but should according to spec
New submission from mike bayer mike...@zzzcomputing.com: Copying this bug from the pysqlite tracker, at http://code.google.com/p/pysqlite/issues/detail?id=21 , as the issue has been opened for two days with no reply. (side node - should sqlite3 bugs be reported here or on the pysqlite tracker ?) The text below was originally written by Randall Nortman: Pysqlite does not open a transaction in the database until a DML statement is encountered (INSERT, UPDATE, or DELETE). A DQL (SELECT) statement will not cause a transaction to be opened if one is not already opened. This is the documented behavior, but it is not what is intended by the spec (PEP 249). The spec intends a transaction to always be open (per the spec author), and this is what happens in other DB-API drivers. For more information, see the this DB-SIG mailing list post (by the PEP 249 author): http://mail.python.org/pipermail/db-sig/2010-September/005645.html For additional background, see this thread on the SQLAlchemy mailing list, which is the source of the attached test case: http://groups.google.com/group/sqlalchemy/browse_thread/thread/2f47e28c1fcdf9e6/0ef1666759ce0724#0ef1666759ce0724 What steps will reproduce the problem? 1. See attached test case. Run it as is, and the final conn1.commit() statement will complete successfully. 2. Uncomment the c2.execute(BEGIN) line and run again; this time conn1.commit() hangs until a timeout, then a Database is locked error is returned. What is the expected output? What do you see instead? The BEGIN should be issued implicitly, and even without doing it explicitly, the commit should block and then return the DB locked error. What version of the product are you using? On what operating system? Python 2.6.6 with its built-in sqlite3 module, on Debian Squeeze x86. import sqlite3 import os if os.path.exists(file.db): os.unlink(file.db) conn1 = sqlite3.connect(file.db) c1 = conn1.cursor() c1.execute(PRAGMA read_uncommitted=SERIALIZABLE) c1.execute(create table foo (id integer primary key, data varchar(30))) c1.execute(insert into foo(id, data) values (1, 'data1')) c1.close() conn1.commit() c1 = conn1.cursor() c1.execute(select * from foo where id=1) row1 = c1.fetchone() c1.close() conn2 = sqlite3.connect(file.db) c2 = conn2.cursor() c2.execute(PRAGMA read_uncommitted=SERIALIZABLE) # sqlite3 should be doing this automatically. # when called, conn1's commit blocks #c2.execute(BEGIN) c2.execute(select * from foo where id=1) row2 = c2.fetchone() c2.close() c1 = conn1.cursor() c1.execute(update foo set data='data2') print About to commit conn1... conn1.commit() -- components: Library (Lib) messages: 117167 nosy: zzzeek priority: normal severity: normal status: open title: sqlite3 SELECT does not BEGIN a transaction, but should according to spec type: behavior versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9924 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9924] sqlite3 SELECT does not BEGIN a transaction, but should according to spec
mike bayer mike...@zzzcomputing.com added the comment: My own comment here is that I'm supposing the late BEGIN behavior is to cut down on SQLite's file locking.I think a way to maintain that convenience for most cases, while allowing the stricter behavior that makes SERIALIZABLE isolation worthwhile, would be an option to sqlite3.connect() that moves the implicit BEGIN to before any DQL, not just DML, statement. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9924 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9405] crash when calling urllib.getproxies() under OSX with subprocess / particular memory usage
mike bayer mike...@zzzcomputing.com added the comment: Yeah we are fine if I build the release2.7-maint branch.Also just noticed that this bug prevents really basic things, like using urllib2 from a Pylons application that's running in daemon mode. Pretty major, though unusual that I have some colleagues who don't have the issue with 2.6. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9405 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9405] crash when calling urllib.getproxies() under OSX with subprocess / particular memory usage
New submission from mike bayer mike...@zzzcomputing.com: I'm not optimistic that this will be reproducible elsewhere. I get a silent failure with 2.6 and a crash dialog with 2.7 with the following script. All elements are necessary, although the pkg_resources import may be arbitrary, and reproduces if it comes from the latest setuptools or if it comes from Distribute.I've attached the crash report generated by OSX in the event that someone knows how to interpret it. The script fails for me every time, though not for other people I've had test. import pkg_resources import urllib import multiprocessing def sub(): print about to call getproxies ! urllib.getproxies() print getproxies has completed ! process = multiprocessing.Process(target=sub) process.start() print hi! process.join() Process: Python [79964] Path: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: ??? (???) Code Type: X86 (Native) Parent Process: Python [79963] Date/Time: 2010-07-28 17:24:19.281 -0400 OS Version: Mac OS X 10.6.4 (10F569) Report Version: 6 Interval Since Last Report: 1137309 sec Crashes Since Last Report: 87 Per-App Crashes Since Last Report: 86 Anonymous UUID: 848E03B6-EF61-45EA-9BB7-BF9E584CC670 Exception Type: EXC_BAD_ACCESS (SIGBUS) Exception Codes: KERN_PROTECTION_FAILURE at 0x0104 Crashed Thread: 0 Dispatch queue: CFMachPort Thread 0 Crashed: Dispatch queue: CFMachPort 0 libSystem.B.dylib 0x94419bdd _dispatch_wakeup + 91 1 libSystem.B.dylib 0x9454a1f2 _dispatch_queue_push_list_slow + 37 2 libSystem.B.dylib 0x94419c22 _dispatch_wakeup + 160 3 libSystem.B.dylib 0x944196e6 _dispatch_source_create2 + 194 4 libSystem.B.dylib 0x94419278 dispatch_source_timer_create_f + 286 5 libSystem.B.dylib 0x94419158 dispatch_source_timer_create + 81 6 com.apple.CoreFoundation0x92fde72a ___CFMachPortCreateWithPort2_block_invoke_1 + 762 7 libSystem.B.dylib 0x944190c3 dispatch_barrier_sync_f + 78 8 libSystem.B.dylib 0x94418f9d dispatch_sync + 48 9 com.apple.CoreFoundation0x92fde350 _CFMachPortCreateWithPort2 + 272 10 com.apple.CoreFoundation0x92fe5e1c CFMachPortCreate + 204 11 com.apple.CoreFoundation0x92fe5c39 _CFXNotificationCenterCreate + 281 12 com.apple.CoreFoundation0x92fe5ad6 _CFXNotificationGetHostCenter + 86 13 com.apple.CoreFoundation0x92feb432 __CFXPreferencesGetSourceForTriplet + 354 14 com.apple.CoreFoundation0x92fea6ad __CFXPreferencesGetSearchListForBundleID + 205 15 com.apple.CoreFoundation0x92fea555 CFPreferencesCopyAppValue + 53 16 com.apple.SystemConfiguration 0x90fc7348 SCDynamicStoreCopyProxies + 44 17 _scproxy.so 0x00746e31 get_proxies + 33 18 org.python.python 0x000ca2b3 PyEval_EvalFrameEx + 20419 19 org.python.python 0x000cbc88 PyEval_EvalFrameEx + 27032 20 org.python.python 0x000cbc88 PyEval_EvalFrameEx + 27032 21 org.python.python 0x000cc4ba PyEval_EvalCodeEx + 2042 22 org.python.python 0x00041ca2 function_call + 162 23 org.python.python 0xf375 PyObject_Call + 85 24 org.python.python 0x000c7d5b PyEval_EvalFrameEx + 10859 25 org.python.python 0x000cbc88 PyEval_EvalFrameEx + 27032 26 org.python.python 0x000cbc88 PyEval_EvalFrameEx + 27032 27 org.python.python 0x000cc4ba PyEval_EvalCodeEx + 2042 28 org.python.python 0x00041ca2 function_call + 162 29 org.python.python 0xf375 PyObject_Call + 85 30 org.python.python 0x00021c66 instancemethod_call + 422 31 org.python.python 0xf375 PyObject_Call + 85 32 org.python.python 0x0007c8a7 slot_tp_init + 87 33 org.python.python 0x0007b2d0 type_call + 176 34 org.python.python 0xf375 PyObject_Call + 85 35 org.python.python 0x000c8ad6 PyEval_EvalFrameEx + 14310 36 org.python.python 0x000cbc88 PyEval_EvalFrameEx + 27032 37 org.python.python 0x000cc4ba PyEval_EvalCodeEx + 2042 38 org.python.python 0x000cc647 PyEval_EvalCode + 87 39 org.python.python 0x000f0ae8 PyRun_FileExFlags + 168 40 org.python.python 0x000f1a23 PyRun_SimpleFileExFlags + 867 41 org.python.python 0x0010a42b Py_Main + 3163 42 org.python.python 0x1f82 0x1000 + 3970 43 org.python.python
[issue9405] crash when calling urllib.getproxies() under OSX with subprocess / particular memory usage
mike bayer mike...@zzzcomputing.com added the comment: right...so I would propose the function calls in question emit a warning or something when called in a child process. this would save lots of people many hours of grief. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9405 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9269] Cannot pickle self-referencing sets
mike bayer mike...@zzzcomputing.com added the comment: where is it defined that sets are not supposed to contain mutable items? such a requirement vastly limits the usefulness of sets. Consider that relational database rows are mutable. A result set containing multiple rows which each have a primary key comprises a set, hashed on primary key. But other attributes of each row can be updated. Surely this is not controversial ? What Python data structure should I (and a whole bunch of Python ORMs) be using to represent mutable, relational database rows, unordered and unique on primary key, in memory ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9269 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9269] Cannot pickle self-referencing sets
mike bayer mike...@zzzcomputing.com added the comment: OK, more specifically, here's the kind of situation where items in a set are mutable: company = Session.query(Company).first() # company.employees is a set() company.employees # each employee references the parent company for e in company.employees: assert e.company is company So nothing is mutated relationally in this case. It's just a plain old bidirectional structure.If two objects are related via many-to-many, then you might have a set in both directions. I'm not sure if this specific situation produces the pickle bug, however, it's been awhile since I've seen which specific case causes the problem. But there are mutable items in a set in these examples and it doesn't seem unreasonable to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9269 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7922] Python 3's 2to3 does not handle non-ascii source files
New submission from mike bayer mike...@zzzcomputing.com:
given the following Python 2 source file:
# -*- encoding: utf-8
print 'bien mangé'
It can be converted to Python 3 using 2's 2to3 tool:
classic$ 2to3 test.py
... omitted ...
--- test.py (original)
+++ test.py (refactored)
@@ -1,3 +1,3 @@
# -*- encoding: utf-8
-print 'bien mangé'
+print('bien mangé')
However that of Python 3.1.1 fails:
classic$ 2to3-3.1 test.py
... omitted ...
--- test.py (original)
+++ test.py (refactored)
@@ -1,3 +1,3 @@
# -*- encoding: utf-8
Traceback (most recent call last):
File /usr/local/bin/2to3-3.1, line 6, in module
sys.exit(main(lib2to3.fixes))
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py,
line 159, in main
options.processes)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py,
line 616, in refactor
items, write, doctests_only)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py,
line 276, in refactor
self.refactor_file(dir_or_file, write, doctests_only)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py,
line 656, in refactor_file
*args, **kwargs)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py,
line 332, in refactor_file
write=write, encoding=encoding)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py,
line 432, in processed_file
self.print_output(old_text, new_text, filename, equal)
File
/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py,
line 64, in print_output
print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position
17: ordinal not in range(128)
--
components: 2to3 (2.x to 3.0 conversion tool)
messages: 99298
nosy: zzzeek
severity: normal
status: open
title: Python 3's 2to3 does not handle non-ascii source files
type: crash
versions: Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7922
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7922] Python 3's 2to3 does not handle non-ascii source files
mike bayer mike...@zzzcomputing.com added the comment: yes, its handled: WARNING: couldn't encode test.py's diff for your terminal is that fix specific to 2to3 or is that just how print works in 3.2 ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7922 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6245] Add intel universal architecture on OSX
Changes by mike bayer mike...@zzzcomputing.com: -- nosy: +zzzeek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6245 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue998998] pickle bug - recursively memoizing class?
mike bayer mike...@zzzcomputing.com added the comment: im noticing my test case seems to work fine in py 3.0.1. haven't tested 2.6.2. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue998998 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6149] WeakValueDictionary constructor ported to Python 3.0 incorrectly
New submission from mike bayer mike...@zzzcomputing.com:
The constructor for WeakValueDictionary does not obey the contract
documented in its comments:
# We inherit the constructor without worrying about the input
# dictionary; since it uses our .update() method, we get the right
# checks
yet it initializes with:
self.data = d = {}
d.update(*args, **kw)
i.e. the update() method of dict, so that a dict passed to the
constructor initializes non-weakrefed values in the dict which is
completely invalid state.
Contrast to that of 2.6, which properly uses the superclass:
UserDict.UserDict.__init__(self, *args, **kw)
A simple test which raises an exception on 3.0.1 is as follows:
import weakref
class Foo(object):
pass
mydict = dict((k, Foo()) for k in range(10))
w = weakref.WeakValueDictionary(mydict)
assert w[5]
--
messages: 88577
nosy: zzzeek
severity: normal
status: open
title: WeakValueDictionary constructor ported to Python 3.0 incorrectly
versions: Python 3.0
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6149
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6149] WeakValueDictionary constructor ported to Python 3.0 incorrectly
Changes by mike bayer mike...@zzzcomputing.com: -- components: +Library (Lib) type: - crash ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6149 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5331] multiprocessing hangs when Pool used within Process
New submission from mike bayer mike...@zzzcomputing.com: this occurs for me running on Mac OSX Leopard. The equivalent code using processing in python 2.5 works fine, which is how I found this bug - my code hung when upgraded to 2.6.Basically initiating a multiprocessing.Pool inside of multiprocessing.Process hangs the application. Below is code which illustrates the issue on both py2.6 and py3.0: from multiprocessing import Pool, Process import sys def f(x): return x*x def go(): pool = Pool(processes=4) x = pool.map(f, [1, 2, 3, 4, 5, 6, 7]) sys.stdout.write(result: %s\n % x) def go2(): x = map(f, [1,2,3,4,5,6,7]) sys.stdout.write(result: %s\n % x) # pool alone, fine go() # process alone, fine p = Process(target=go2) p.start() p.join() # use both, hangs p = Process(target=go) p.start() p.join() -- components: Library (Lib) messages: 82534 nosy: zzzeek severity: normal status: open title: multiprocessing hangs when Pool used within Process type: crash versions: Python 2.6, Python 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5331 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue998998] pickle bug - recursively memoizing class?
mike bayer [EMAIL PROTECTED] added the comment: This bug can be reproduced very easily: import pickle class MyClass(object): pass m = MyClass() m2 = MyClass() s = set([m]) m.foo = set([m2]) m2.foo = s print pickle.dumps(s) This bug is critical as the pure-python pickle module is required when trying to remove warnings in -3 mode with Python 2.6, which is something that will be very common in the near future. It's easily reproducible with simplistic object cycles like that of the above, in 2.5, 2.6 and 3.0 at least. -- nosy: +zzzeek ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue998998 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
