Hi Pierre,
As far as I understand, bfloat16 is specialized for machine learning (and other applications that require little precision but might have inputs of very varying magnitude). The format spec does not state it explicitly, but Arrow expects floating-point data to be represented in IEEE format. If you want to transfer bfloat16 data, you may want to create an extension type for it. The underlying storage type could be UInt16 or FixedSizeBinary(2), for example. (perhaps at some point we may want to create and maintain a reference of "standard" extension types such as UUID, IPv4, etc.) Regards Antoine. Le 09/06/2020 à 01:45, Pierre Belzile a écrit : > Hi, > > There seems to be two competing standards for floats with 16 bits: > > - https://en.wikipedia.org/wiki/Bfloat16_floating-point_format > - IEEE: https://en.wikipedia.org/wiki/IEEE_754-2008_revision > > Was there any thought on how this could be handled? Would it make sense to > add some kind of DataType attribute to the HALF_FLOAT? > > Cheers, Pierre >
