[issue38665] Crash when running SQLAlchemy with pyodbc
Dave Johansen added the comment: I can't. I just know that I'm running this process and this crash happens. Any recommendations on how to diagnose that? -- ___ Python tracker <https://bugs.python.org/issue38665> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38665] Crash when running SQLAlchemy with pyodbc
New submission from Dave Johansen : We're using SQLAlchemy 1.3.10 with pyodbc 4.0.27 in the python:3.7.5-alpine docker image to connect to a MySQL 13.0.5026.0 database and it's crashing with the following error: python: malloc.c:2406: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed. -- messages: 355830 nosy: Dave Johansen priority: normal severity: normal status: open title: Crash when running SQLAlchemy with pyodbc type: crash versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue38665> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37092] LoggerAdapter doesn't have fatal() like Logger
New submission from Dave Johansen : Using LoggerAdapter is a convenient way to add extra info to all logs, but it doesn't have the fatal() method like Logger, so it isn't a drop in replacement like it should be. -- components: Library (Lib) messages: 343941 nosy: Dave Johansen priority: normal severity: normal status: open title: LoggerAdapter doesn't have fatal() like Logger type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue37092> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31212] datetime: min date (0001-01-01 00:00:00) can't be converted to local timestamp
Dave Johansen added the comment: Ok, so I understand the issue now. `timestamp()` for naive datetime instances applies the local timezone offset ( https://docs.python.org/3.6/library/datetime.html#datetime.datetime.timestamp ). This is surprising because naive datetime instances usually are just that and don't make assumptions the timezone, but I guess that the behavior is documented and I was just unaware of it. It still seems like this shouldn't give an error (especially when the timezone of the local machine is UTC), but I guess that it is what it is. -- ___ Python tracker <http://bugs.python.org/issue31212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31212] datetime: min date (0001-01-01 00:00:00) can't be converted to local timestamp
Dave Johansen added the comment: The use case was parsing user input of ISO 8601 date strings and converting them to UNIX epochs. The input "0001-01-01T00:00:00" is valid, parses to a valid `datetime` and it seems like a reasonable expectation that all of the functions should work on a valid `datetime` (especially when they worked in earlier versions of Python). -- ___ Python tracker <http://bugs.python.org/issue31212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31212] datetime: min date (0001-01-01 00:00:00) can't be converted to local timestamp
Dave Johansen added the comment: That's a valid `datetime` (i.e. within the min and max values) and `tzinfo` is `None` so I think it's completely reasonable to assume that `timestamp()` will return the correct value. -- ___ Python tracker <http://bugs.python.org/issue31212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31212] min date can't be converted to timestamp
New submission from Dave Johansen: This worked in Python 3.6.0 and before: ``` from datetime import datetime d = datetime(1, 1, 1, 0, 0, 0) d.timestamp() ``` The error output is: ``` ValueError: year 0 is out of range ``` But it used to return `-62135658000.0`. Appears to be related to https://bugs.python.org/issue29921 -- messages: 300303 nosy: Dave Johansen priority: normal severity: normal status: open title: min date can't be converted to timestamp type: behavior versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue31212> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18426] Crash when extension does not use PyModule_Create()
Ivan Johansen added the comment: Probably not. I am setting status to closed with resolution fixed. -- resolution: -> fixed status: pending -> closed ___ Python tracker <http://bugs.python.org/issue18426> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18426] Crash when extension does not use PyModule_Create()
Ivan Johansen added the comment: If possible it would be nice if any module could be returned from a C extension. Specifically I was trying to subclass module (PyModule_Type) and use that. But an error message is better than a crash. -- ___ Python tracker <http://bugs.python.org/issue18426> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18426] Crash when extension does not use PyModule_Create()
New submission from Ivan Johansen: In Python/importdl.c around line 99 in the function _PyImport_LoadDynamicModule() you can find the code: def = PyModule_GetDef(m); def->m_base.m_init = p; If the module m, which is returned from a newly imported extension, is not created by PyModule_Create() but in some other way then PyModule_GetDef(m) will return NULL. The next line will then dereference a NULL pointer and crash. I suggest a check for this is added: def = PyModule_GetDef(m); if(def != NULL) def->m_base.m_init = p; -- messages: 192845 nosy: Padowan priority: normal severity: normal status: open title: Crash when extension does not use PyModule_Create() type: crash versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <http://bugs.python.org/issue18426> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9811] strftime strips '%' from unknown format codes on OS X
Sverre Johansen added the comment: This is because of differences in GNU and BSD C stdlib; I get the same results using the BSD and GNU versions of `date`. $ date +"%q" What does Python typically do in cases like this? -- ___ Python tracker <http://bugs.python.org/issue9811> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9811] strftime strips '%' from unknown format codes on OS X
New submission from Sverre Johansen :
There seems to be a platform difference in the way stftime handles
unknown format codes.
In OSX Python removes the percentage sign from the returned string when the
format code is unknown. In Linux it leaves it.
Look at the following example:
This is Python 3.1.2 in Ubuntu:
Python 3.1.2 (r312:79147, Apr 15 2010, 15:35:48)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> datetime.datetime.now().strftime("%q")
'%q'
And this is Python 3.1.2 on Max OSX:
Python 3.1.2 (r312:79147, Sep 9 2010, 11:11:24)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> datetime.datetime.now().strftime("%q")
'q'
I've gotten the same result in 2.5, 2.6 and 3.1.
FYI, this broke some code that parsed the same string first with strftime, and
then used string interpolation (The mac version
deleted all percentage signs before '('):
>>> datetime.datetime.now().strftime("%Y - %(foobar)s") % {"foobar": "egg"}
This does not work in OSX.
--
components: Library (Lib)
messages: 115933
nosy: sverrejoh
priority: normal
severity: normal
status: open
title: strftime strips '%' from unknown format codes on OS X
type: behavior
versions: Python 3.1
___
Python tracker
<http://bugs.python.org/issue9811>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2159] dbmmodule inquiry function is performance prohibitive
johansen added the comment: I haven't been able to find any of the patches listed in the comments, but it does look like providing a nb_nonzero method in the module would solve our issue. PyObject_IsTrue checks the tp_as_number methods before the sequence and mapping methods. I'm not sure if it's safe to count on this behavior as always being true, but for 2.4 and the dbmmodule, it would work. ___ Python tracker <http://bugs.python.org/issue2159> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3014] file_dealloc() assumes errno is set when EOF is returned
johansen <[EMAIL PROTECTED]> added the comment: As I said before: check_and_flush calls ferror(3C) and then fflush(3C) on the FILE stream associated with the file object. There's just one problem here. If it finds an error that was previously encountered on the file stream, there's no guarantee that errno will be valid. Should an error be encountered in fflush(3C), errno will get set; however, the contents of errno are undefined should fflush() return successfully. The problem is this: The caller of check_and_flush() will check errno if check_and_flush returns an error. However, check_and_flush() checks two different error sources. If ferror() returns an error, you can't check errno, because that routine doesn't set errno. However, if fflush() returns an error, it will set errno. In its current form, no caller of check_and_flush() should check errno if the check_and_flush doesn't return successfully, since there's no way of knowing whether errno will be set or not. That doesn't make a lot of sense to me, so I would probably re-write this code. However, since I'm not an expert in the inner workings of the Python interpreter, it's hard for me to suggest just how to do this. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3014> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3014] file_dealloc() assumes errno is set when EOF is returned
johansen <[EMAIL PROTECTED]> added the comment: The problem is present in Python 2.4.4, the version that we're using here. I'm not familiar with the versions keyword that's used here, but that's the version for which I'm reporting this bug. To be clear, the problem is that when file_dealloc() sees an EOF, it assumes that an error has occurred. It then checks errno. However, it's possible for file_dealloc() to get an EOF under conditions where errno hasn't been set. In that case, it reports an error with a stale value of errno. This is explained in detail in the initial report. I don't think the title is misleading; it's incorrect for file_dealloc() to assume that errno is set every time it gets an EOF. -- versions: +Python 2.4 -Python 2.7 ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3014> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2650] re.escape should not escape underscore
Morten Lied Johansen <[EMAIL PROTECTED]> added the comment: In my particular case, we were passing the regex on to a database which has regex support syntactically equal to Python, so it seemed natural to use re.escape to make sure we weren't matching against the pattern we really wanted. The documentation of re.escape also states that it will only escape non- alphanumeric characters, which is apparently only true if you are using a single byte encoding (ie. not utf-8, or any other encoding using more than a single byte per character). At the very least, that's probably worth mentioning in the docs. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2650> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2650] re.escape should not escape underscore
Morten Lied Johansen <[EMAIL PROTECTED]> added the comment: One issue that the current implementation has, which I can't see have been commented on here, is that it kills utf8 characters (and probably every other character encoding that is multi-byte). A é character in an utf8 encoded string will be represented by two bytes. When passed through re.escape, those two bytes are checked individually, and both are considered non-alphanumeric, and is consequently escaped, breaking the utf8 string into complete gibberish instead. -- nosy: +mortenlj ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2650> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3196] Option in pydoc to show docs from private methods
New submission from Morten Lied Johansen <[EMAIL PROTECTED]>: Currently, running pydoc on a module will show you the documentation on all regular methods, and all special methods (starting and ending in double underscores). Private methods (starting with a single underscore) are not included. Some times, it would be nice to include docs for these methods aswell, and a small change to pydoc.visiblename solves the problem. I've included a patch that adds this behaviour as an option (-i for include private names). The implementation isn't as clean as one would hope for (sets a global flag), but was the best I could come up with in the short time I had available. Feel free to make a better implementation. The patch is against python 2.5. -- components: Library (Lib) files: pydoc.privatenames.patch keywords: patch messages: 68722 nosy: mortenlj severity: normal status: open title: Option in pydoc to show docs from private methods type: feature request versions: Python 2.5 Added file: http://bugs.python.org/file10727/pydoc.privatenames.patch ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3196> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2159] dbmmodule inquiry function is performance prohibitive
johansen <[EMAIL PROTECTED]> added the comment: Yes, True/False should be sufficient for our purposes. IIRC, we were trying to determine if we had a stale handle to the database and needed to open it again. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2159> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3014] file_dealloc() assumes errno is set when EOF is returned
New submission from johansen <[EMAIL PROTECTED]>:
We're using Python to build the new packaging system for OpenSolaris.
Yesterday, a user reported that when they ran the pkg command, piped the
output to grep, and then typed ^C, sometimes they'd get this error:
$ pkg list | grep office
^Cclose failed: [Errno 11] Resource temporarily unavailable
We assumed that this might be a problem in the signal handling we've
employed to catch SIGPIPE; however, it turns out that the problem is in
the file_dealloc() code.
For the perversely curious, additional details may be found in the
original bug located here:
http://defect.opensolaris.org/bz/show_bug.cgi?id=2083
Essentially we found the following:
The error message is emitted from fileobject.c: file_dealloc()
The relevant portion of the routine looks like this:
static void
file_dealloc(PyFileObject *f)
{
int sts = 0;
if (f->weakreflist != NULL)
PyObject_ClearWeakRefs((PyObject *) f);
if (f->f_fp != NULL && f->f_close != NULL) {
Py_BEGIN_ALLOW_THREADS
sts = (*f->f_close)(f->f_fp);
Py_END_ALLOW_THREADS
if (sts == EOF)
#ifdef HAVE_STRERROR
PySys_WriteStderr("close failed: [Errno %d] %s\n",
errno, strerror(errno));
In the cases we encountered, the function pointer f_close is actually a
call to sysmodule.c: _check_and_flush()
That routine looks like this:
static int
_check_and_flush (FILE *stream)
{
int prev_fail = ferror (stream);
return fflush (stream) || prev_fail ? EOF : 0;
}
check_and_flush calls ferror(3C) and then fflush(3C) on the FILE stream
associated with the file object. There's just one problem here. If it
finds an error that was previously encountered on the file stream,
there's no guarantee that errno will be valid. Should an error be
encountered in fflush(3C), errno will get set; however, the contents of
errno are undefined should fflush() return successfully.
Here's what happens in the code I observed:
I set a write watchpoint on errno and observed the different times it
was accessed. After sifting through a bunch of red-herrings, I found
that a call to PyThread_acquire_lock() that sets errno to 11 (EAGAIN).
This occurs when PyThread_acquire_lock() calls sem_trywait(3C) and finds
the semaphore already locked. Errno doesn't get accessed again until a
call to libc.so.1`isseekable() that simply saves and restores the
existing errno.
Since we've taken a ^C (SIGINT), the interpreter begins the finalization
process and eventually calls file_dealloc(). This routine calls
_check_and_flush(). In the case that I observed, ferror(3C)
returns a non-zero value but fflush(3C) completes successfully. This
causes the routine to return EOF to the caller. file_dealloc() assumes
that since it received an EOF an error occurred and it should call
strerror(errno). However, since this is just returning the state of a
previous error, errno is invalid.
This is what causes the spurious EAGAIN message. Just to be sure, I
traced the return value and errno of failed syscalls that were invoked
by the interpreter. I was unable to observe any syscalls returning
EAGAIN. This is because (at least on OpenSolaris) sem_trywait(3C) calls
sema_trywait(3C). The sema_trywait returns EBUSY if the semaphore is
held and sem_trywait converts this to EAGAIN. None of these errors are
passed out of the kernel.
It's not clear to me whether _check_and_flush(), file_dealloc(), or both
need modification. At a minimum, it's not safe for file_dealloc() to
assume that errno is set correctly if the function underneath it is
using ferror(3C) to find the presence of an error on the stream.
--
components: Interpreter Core
messages: 67560
nosy: johansen
severity: normal
status: open
title: file_dealloc() assumes errno is set when EOF is returned
type: behavior
versions: Python 2.4
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue3014>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2159] dbmmodule inquiry function is performance prohibitive
New submission from johansen:
We've been using Python 2.4 to build the new package management software
for OpenSolaris. We use a ndbm database to hold keywords about
packages, and found that each time we added a new OpenSolaris build to
our package repository, the time to import would increase by about 1/3
of the previous time.
It turns out that we were continually invoking a function in the
dbmmodule that walked the entire database every time the function was
called.
Looking at dbmmodule.c, the source for dbm.so, is instructive:
This is dbm_length, the function that we're _always_ calling.
static int
dbm_length(dbmobject *dp)
{
if (dp->di_dbm == NULL) {
PyErr_SetString(DbmError, "DBM object has already been
closed");
return -1;
}
if ( dp->di_size < 0 ) {
datum key;
int size;
size = 0;
for ( key=dbm_firstkey(dp->di_dbm); key.dptr;
key = dbm_nextkey(dp->di_dbm))
size++;
dp->di_size = size;
}
return dp->di_size;
}
It's a knock-off of function shown in ndbm(3C) that traverses the
database. It looks like this function walks every record in the
database, and then returns that as its size.
Further examination of dbmmodule shows that dbm_length has been assigned
as the function for the inquiry operator:
static PyMappingMethods dbm_as_mapping = {
(inquiry)dbm_length,/*mp_length*/
(binaryfunc)dbm_subscript, /*mp_subscript*/
(objobjargproc)dbm_ass_sub, /*mp_ass_subscript*/
};
It looks like dbm_length stashes the size of the database, so it doesn't
always have to traverse it. However, further examination of the source
shows that an insertion doesn't update the di_size counter. Worse yet,
an update or a delete will cause the counter to be set to -1. This
means that the next call to dbm_length will have to walk the entire
database all over again. Ick.
One of the problem parts of the code is this line in catalog.py:
update_searchdb():
if fmri_list:
if not self.searchdb:
self.searchdb = \
dbm.open(self.searchdb_file, "c")
This if not triggers the PyObject_IsTrue that invokes the inquiry operator
for the dbm module. Every time we run this code, we're going to walk
the entire database. By changing this to:
if fmri_list:
if self.searchdb is None:
self.searchdb = \
dbm.open(self.searchdb_file, "c")
We were able to work around the problem by using the is None check,
instead of if not self.searchdb; however, this seems like it is really a
problem with the dbmmodule and should ultimately be resolved there.
--
components: Extension Modules
messages: 62668
nosy: johansen
severity: normal
status: open
title: dbmmodule inquiry function is performance prohibitive
type: resource usage
versions: Python 2.4
__
Tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue2159>
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
