On Wed, Jul 16, 2014 at 3:48 AM, Todd <toddr...@gmail.com> wrote: > On Jul 16, 2014 11:43 AM, "Chris Barker" <chris.bar...@noaa.gov> wrote: > > So numpy should have dtypes to match these. We're a bit stuck, however, > because 'S' mapped to the py2 string type, which no longer exists in py3. > Sorry not running py3 to see what 'S' does now, but I know it's bit broken, > and may be too late to change it > > In py3 a 'S' dtype is converted to a python bytes object. > right -- thanks. That's the source of the problems.
A bit of a higher-level view of the issues at hand. Python has three relevant data types: A unicode type (unicode in py2, str in py3) A one-byte-per-char stringtype (py2 string) A bytes type The big problem is that py2 only has the unicode and py2string types, and py3 only has the unicode and bytes type. numpy has 'S' and 'U' types: which map naturally to the py2string and unicode types. but since py3 has no py2string type, we have a problem. If numpy were to embrace the py3 model, then 'S' should have mapped to py3's string, aka unicode. But: 1) then there would be no bytes type, which is a problem, as people do need to a pass collections of bytes around. I"ve alwyas figured numpy's uint8 should suffice for that, but "strings of bytes" are useful, and it seem to be awkward, or maybe impossible to construct such a beast with the usual dtype machinery 2) there is a need (or at least a desire), to have a compact, one-byte-per-charater text type in numpy. Thinking of it in this framework leads me to the conclusion that numpy should have three types: 1) A unicode type --no change here 2) A bytes types -- almost the current 'S' type - A bytes type would map to/from py3 bytes objects (and py2 bytes objects, which are the same as py2strings) - one way is would differ from a py2str is that there would be no assumption of null-termination (not sure where that is now) 3) A one-byte-per-char text type -- more or less Chuck's current proposal. - it would map to/from the py3 string -- it is text after all - it would be null-terminated - it would have a one-byte per-char encoding: ascii, latin-1 or settable (TBA) It would be nice if numpy had built-in encoding/decoding to/from the unicode type to/from the bytes type (tricky due to not knowing how many bytes a given string will decode to without decoding it.. Which leaves us with the decisions: * what does 'S' map to? - currently it's almost a bytes type, and maps to bytes in py3 -- so maybe keep that status quo. Except that it really doesn't act like text anymore, so 2 to 3 transition is kind of ugly, and the name is misleading. * what encoding to use for the one-byte-per-char-text-type? - I think latin-1 is the way to go -- you could use it like asciii if you want, but if you need a few other characters they are there. And you can even store binary data in it, thought that's a "bad idea" anyway. - ascii would solve common use cases, but I see no reason to restrict folks to 127 characters -- you can use those if you like. If the binary data needs to get passed to something that really needs to be ascii-only, it could be checked at that point. - perhaps the best option is for client code to be able chose an encoding -- but more code, maybe a more confusing interface? worth it? * Do we have a utf-8 type?: I think not -- it simply does not map to both unicode and numpy's fixed-length requirement. If all this gets done, we have some transition issues, but I think it would solve everyone's problems (though maybe not as cleanly as we'd like...). For instance, if someone needs to map numpy arrays to utf-8 data (i.e. HDF5), then they can either use the bytes type and let the user decode, or encode/decode to unicode on i/o. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion