[issue6071] no longer possible to hash arrays

[Benjamin Peterson]

Benjamin Peterson  added the comment:

Done in r77313.

--
nosy: +benjamin.peterson
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Ezio Melotti]

Ezio Melotti  added the comment:

Even if it's fixed a patch with tests should be submitted before closing this 
issue.

--
nosy: +ezio.melotti
priority: deferred blocker -> normal
stage: needs patch -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[ivank]

ivank  added the comment:

I believe this was fixed in r77252, which was fixing 
http://bugs.python.org/issue3745

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jean-Paul Calderone]

Jean-Paul Calderone  added the comment:

Will this be fixed for Python 2.7 final?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> As for the checks for bf_releasebuffer: I still think they are
> necessary. If an object implements bf_releasebuffer, that means that the
> object may change the buffer underneath, unless proper locking and
> unlocking takes place.

I know, but the problem is that by switching some argument definitions
to "s*" and friends we have broken compatibility for the (admittedly
uncommon) use case of giving an array object to those functions. Since
we probably don't want to backout those changes perhaps adding support
for the new buffer API to the array object is the best course of action.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Martin v . Löwis]

Martin v. Löwis  added the comment:

I think the error is really in _hashlib, not in the array object. It
should not require 3.x style buffers, but continue to support 2.x
readbuffers. Attached is a patch that takes this route to fixing the bug.

As for the checks for bf_releasebuffer: I still think they are
necessary. If an object implements bf_releasebuffer, that means that the
object may change the buffer underneath, unless proper locking and
unlocking takes place. Indeed, the array's getreadbuf operation is not
thread-safe. It might be possible to remove them if it is clarified that
anybody calling getreadbuffer must not release the GIL while they hold
on to the buffer.

--
Added file: http://bugs.python.org/file15382/hashlib.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I would prefer if I opened a separate issue for this bug.

if /you/ opened, of course...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I fixed the array_ass_slice for python 2.7 in the attached patch.

I would prefer if I opened a separate issue for this bug. That way it
will be easier to port the patch separately to py3k.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

I fixed the array_ass_slice for python 2.7 in the attached patch.

The problem should apply to python 3k as well: firstly the above which 
might stay unnoticed and secondly the function moves to much memory if the 
array size is increased: (Py_SIZE(a)-d-ihigh) items should be moved, 
because Py_SIZE(a) was already modified by array_resize, but the function 
moves (Py_SIZE(a)-ihigh) items.

The attached patch for python 2.7 passes all unit tests for me.

--
Added file: http://bugs.python.org/file14999/hasharray.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

I was looking at the remaining differences between Modules/arraymodule.c 
in python 2.7 and the 3k branch, in particular I was testing why 
including the changes to the function array_ass_slice into the patch 
breaks the unit test of the array module.

The manual resizing of the memory basically was replaced by a call to 
array_resize with the advantage, that checking for exported memory views 
is done there (which is not necessary because it's already done in the 
function) and the code gets clearer. I think when PyMem_RESIZE is called 
the pointer to the memory might change. So in 3k this now happens in 
array_resize, so the array->ob_item pointer changes but not it's local 
copy in array_ass_slice. Isn't that potentially causing trouble in 
python 3k?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Moving to deferred blocker to get this out of the way for 2.6.3

--
nosy: +barry
priority: release blocker -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

After thinking about it, we should remover the bf_releasebuffer checks
when using the old typecodes '#' (the doc should also clearly state that
these are unsafe and '*' is preferred).

If a type chooses to implement the /old/ buffer API and if a method
requires the /old/ buffer API instead of the new one (that is, by using
a '#' typecode), we have to accept these choices.

Here's a patch for trunk, combining the new buffer API for `array` and
the checks removal in getargs.c.

--
Added file: http://bugs.python.org/file14995/hasharray.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> The first thing the function does is checking if the object implements 
> the old buffer api, but also fails if pb->bf_releasebuffer != NULL. So I 
> guess it's also making sure the new buffer api is not implemented.
> 
> What's the thought behind this?

The idea is that if a type implements the bf_releasebuffer function, it
shouldn't be used in a situation where the caller won't try to release
the buffer.

It looks a bit unwarranted though, because a type implementing the old
buffer API should be able to function without the releasing anyway.
Otherwise it wouldn't implement that API at all.

Martin, ISTR you did that addition, would you be opposed to removing the
NULL check on bf_releasebuffer when trying to get a buffer through the
old buffer API?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> The patch breaks five unit tests from other modules, I'll look into it.

What are those tests?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

I stumbled upon the following function:

static Py_ssize_t
convertbuffer(PyObject *arg, void **p, char **errmsg)
in Python/getargs.c

The first thing the function does is checking if the object implements 
the old buffer api, but also fails if pb->bf_releasebuffer != NULL. So I 
guess it's also making sure the new buffer api is not implemented.

What's the thought behind this? Removing that condition fixes three of 
the failing tests but breaks none.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

You asked "what" not "which" :)

test test_codecs failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_codecs.py", line 531, in 
test_array
codecs.readbuffer_encode(array.array("c", "spam")),
TypeError: must be string or read-only buffer, not array.array


test test_ioctl failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_ioctl.py", line 34, in 
test_ioctl_mutate
r = fcntl.ioctl(tty, termios.TIOCGPGRP, buf, 1)
TypeError: ioctl requires a file or file descriptor, an integer and 
optionally an integer or buffer argument


test test_multiprocessing failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_multiprocessing.py", 
line 1269, in test_connection
self.assertEqual(conn.send_bytes(arr), None)
TypeError: must be string or read-only buffer, not array.array


test test_socket failed -- errors occurred; run in verbose mode for 
details
(have to recheck)


test test_struct failed -- Traceback (most recent call last):
  File "/Users/jan/src/python-svn/Lib/test/test_struct.py", line 468, in 
test_unpack_with_buffer
value, = struct.unpack('>I', data)
error: unpack requires a string argument of length 4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

> > The patch breaks five unit tests from other modules, I'll look into
> > it.
> 
> What are those tests?

test_codecs, test_ioctl, test_multiprocessing, test_socket and 
test_struct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

The patch breaks five unit tests from other modules, I'll look into it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

I added the two functions for the new buffer API. Having an exported 
memory view needs some handling elsewhere, so the array does not change. I 
also added tests for checking that behaviour.

Mainly I copypasted code from py3k which involved redirecting to 
array_resize() instead of doing it manually and doing the checking there.

--
keywords: +patch
Added file: http://bugs.python.org/file14980/array_new_buffer.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I tried to implement the new buffer API, but as soon as I add 
> bf_getbuffer/bf_releasebuffer to PyBufferProcs writing an array to a file 
> breaks:

You should take a look at the array module in py3k, it supports the new
buffer API. With a bit of luck, the two relevant functions shouldn't be
too difficult to backport to the trunk.

By the way, to signal that it supports the new buffer API, you must add
Py_TPFLAGS_HAVE_NEWBUFFER to the type flags.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jan Hosang]

Jan Hosang  added the comment:

I tried to implement the new buffer API, but as soon as I add 
bf_getbuffer/bf_releasebuffer to PyBufferProcs writing an array to a file 
breaks:
  f.write(a)
  TypeError: must be contiguous buffer, not array.array

I searched through the file functions, but couldn't find the point where 
this happens. Has anybody a suggestion? Does file.write() use the old 
buffers? Doesn't it use reprfunc?

--
nosy: +chuck

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Georg Brandl]

Georg Brandl  added the comment:

Can't hurt :)

--
nosy: +georg.brandl
priority: high -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jean-Paul Calderone]

Jean-Paul Calderone  added the comment:

Can this ticket be marked as a release blocker so it's not forgotten
about for 2.7?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Martin v . Löwis]

Martin v. Löwis  added the comment:

> It no longer works with objects created with buffer() either:

That's because the buffer objects don't implement the buffer protocol
(in particular, they don't define the Py_TPFLAGS_HAVE_NEWBUFFER flag,
and the bf_getbuffer/bf_releasebuffer operations that go with that flag.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[ivank]

ivank  added the comment:

It no longer works with objects created with buffer() either:

>>> hashlib.sha1(buffer("x")).hexdigest()
Traceback (most recent call last):
  File "", line 1, in 
TypeError: object supporting the buffer API required

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Martin v. Löwis]

Martin v. Löwis  added the comment:

exarkun: would you like to propose a patch?

--
nosy: +loewis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Gregory P. Smith]

Gregory P. Smith  added the comment:

to confirm, this is only a problem in 2.7 trunk right?

if so, i won't rush a fix.

but yes fallback code for legacy types or adding buffer api support to 
array would work.

--
assignee:  -> gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jean-Paul Calderone]

Jean-Paul Calderone  added the comment:

Perhaps Gregory has some idea about how this can most easily be
resolved, since I think he did the work on #3745 which introduced this
change in behavior.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jean-Paul Calderone]

Jean-Paul Calderone  added the comment:

I would certainly like to, but unfortunately at present I am unable to.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Antoine Pitrou]

Antoine Pitrou  added the comment:

That's probably because hashlib switched to using the new buffer API,
and array.array() only supports the old one in 2.x.

Perhaps hashlib should have some fallback code for such legacy types...

--
nosy: +pitrou
priority:  -> high
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[ivank]

Changes by ivank :


--
nosy: +ivank

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6071] no longer possible to hash arrays

[Jean-Paul Calderone]

New submission from Jean-Paul Calderone :

It used to be possible to use hashlib with arrays; it no longer seems
possible.

exar...@charm:~$ python -c '
import sys, hashlib, array
print sys.version_info
print hashlib.sha1(array.array("b", [1, 2, 3])).hexdigest()
'
(2, 5, 2, 'final', 0)
7037807198c22a7d2b0807371d763779a84fdfcf
exar...@charm:~$ ~/Projects/python/trunk/python -c '
import sys, hashlib, array
print sys.version_info
print hashlib.sha1(array.array("b", [1, 2, 3])).hexdigest()
'
sys.version_info(major=2, minor=7, micro=0, releaselevel='alpha', serial=0)
Traceback (most recent call last):
  File "", line 4, in 
TypeError: object supporting the buffer API required
exar...@charm:~$

--
components: Library (Lib)
messages: 88122
nosy: exarkun
severity: normal
status: open
title: no longer possible to hash arrays
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com