On Wed, Jan 22, 2014 at 12:07:28PM -0800, Chris Barker wrote:
> On Wed, Jan 22, 2014 at 2:46 AM, Oscar Benjamin
> <oscar.j.benja...@gmail.com>wrote:
>
> > BTW, as much as the fixed-width 'S' dtype doesn't really work for str in
> > Python 3 it's also a poor fit for bytes since it strips trailing nulls:
> >
> > >>> a = np.array(['a\0s\0', 'qwert'], dtype='S')
> > >>> a
> > array([b'a\x00s', b'qwert'],
> > dtype='|S5')
> > >>> a[0]
> > b'a\x00s'
>
>
> WHOOA! Good catch, Oscar.
>
> This conversation started with me suggesting that 'S' on py3 should mean
> "ascii string" (or latin-1 string).
>
> Then it was pointed out that it was already being used for arbitrary bytes,
> and thus could not be changed to mean a string without breaking already
> working code.
>
> However, if 'S' is assigning meaning to null bytes, and doing something
> with that, then it is, indeed being treated as an ANSI string (or the old c
> string "type", anyway). And any code that is expecting it to be arbitrary
> bytes is already broken, and in a way that could result in pretty subtle,
> hard to find bugs in the future.
>
> I think we really need a proper bytes dtype (which could be 'S' with the
> null byte thing removed), and a proper one-byte-per-character string type.
It's not safe to stop removing the null bytes. This is how numpy determines
the length of the strings in a dtype='S' array. The strings are not
"fixed-width" but rather have a maximum width. Aything shorter gets padded
with nulls. This is transparent if you index strings from the array:
>>> a = np.array(b'a string of different length words'.split(), dtype='S')
>>> a
array([b'a', b'string', b'of', b'different', b'length', b'words'],
dtype='|S9')
>>> a[0]
b'a'
>>> len(a[0])
1
>>> a.tostring()
b'a\x00\x00\x00\x00\x00\x00\x00\x00string\x00\x00\x00of\x00\x00\x00\x00\x00\x00\x00differentlength\x00\x00\x00words\x00\x00\x00\x00'o
If the trailing nulls are not removed then you would get:
>>> a[0]
b'a\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> len(a[0])
9
And I'm sure that someone would get upset about that.
> Though I still don't know the use case for the fixed-length bytes type that
> can't be satisfied with the other numeric types,
Having the null bytes removed and a str (on Py2) object returned is precisely
the use case that distinguishes it from np.uint8. The other differences are the
removal of arithmetic operations.
Some more oddities:
>>> a[0] = 1
>>> a
array([b'1', b'string', b'of', b'different', b'length', b'words'],
dtype='|S9')
>>> a[0] = None
>>> a
array([b'None', b'string', b'of', b'different', b'length', b'words'],
dtype='|S9')
>>> a[0] = range(1, 2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: cannot set an array element with a sequence
>>> a[0] = (x for x in range(2))
>>> a
array([b'<generato', b'string', b'of', b'different', b'length', b'words'],
dtype='|S9')
Oscar
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
