Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
On Thu, May 28, 2009 at 11:39 AM, John Van Enk wrote:
> Writing instances encode/decode that use either little endian or big endian
> (or mixed!) is trivial... see attached file.
>
> I don't see where your problem is...
>
Inexperience, lack of enough contiguous time to digest everything at once
(till now), thank you for your most detailed example! This should be
wiki'd.
Dave
>
> On Thu, May 28, 2009 at 12:35 PM, David Leimbach wrote:
>
>>
>>
>> On Thu, May 28, 2009 at 9:17 AM, John Van Enk wrote:
>>
>>>
Fair enough. I am just new to the interface, wondering if I should
try matching responses by pulling apart via Get, or the bit syntax
package.
>>>
>>> I'm assming you have some 'data Foo = ...'? If this is the case, you're
>>> probably okay writing an instance of Binary for Foo and using encode/decode.
>>> That's all I've had to do and had superb performance.
>>>
>>>
>>
>> encode/decode do Big Endian, and 9P does little endian.
>>
>> From the man page:
>>
>> "Each message consists of a sequence of bytes. Two–, four–, and
>> eight–byte fields hold unsigned integers represented in little–endian order
>> (least significant byte first)."
>>
>> encode/decode just won't work for me as a result, as they assume the
>> not-so-aptly-named (at least in this case) "network" byte order. Since I'm
>> not Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you
>> why they chose little endian for this.
>>
>> (I should note that I sometimes work with CAN hardware, and depending on
>> what people want to put on the line, little endian creeps in there too, so
>> this issue will have to be dealt with here as well, if I am to use
>> Data.Binary)
>>
>> I've got (typing from memory, forgive syntax errors)
>>
>> module Nine (Message) where
>>
>> {-- missing imports --}
>>
>> notag = putWord16le -1
>>
>> -- Message type field values
>> -- from: http://plan9.bell-labs.com/sources/plan9/sys/include/fcall.h
>> tversion = 100
>> rversion = 101
>> rerror = 107
>>
>>
>> -- only 9P2000 is being handled so some constructors are simple
>> data Message = Tversion
>> | Rversion {size :: Int, tag :: Int, msize ::
>> Int, version :: String}
>> | Rerror {message :: String} -- maybe this should
>> be Maybe ByteString?
>> ...
>>
>> instance Binary Message where
>> put Tversion = putWord32le 16 >> tversion >> notag >> msize >>
>> "9P2000"
>> put Rversion = putWord32le 16 >> rversion >> notag >> msize >>
>> "9P2000"
>> get = do size <- getWord32le
>> tag <- getWord8
>> case tag of
>> rversion -> do msize <- getWord32le
>>version <- getRemainingLazyByteString
>>return (Rversion size tag msize
>> (C.unpack version))
>>
>> ===
>>
>> I'm beginning to feel the Get Monad is not going to be as nice as if I had
>> some bit-syntax for determining what type of return message I'm getting.
>> Any "T" message can be met with the paired "R" response, or an Rerror
>> message, with a detailed description of what went wrong. My instance of get
>> is looking like it's going to be pretty nasty :-)
>>
>> The first message in any 9p session is Tversion with Rversion or Rerror
>> response. There's optional authentication hooks via Tauth/Rauth, and then
>> the filesystem namespace navigation operation messages.
>>
>> Dave
>>
>> /jve
>>>
>>
>>
>> ___
>> Haskell-Cafe mailing list
>> Haskell-Cafe@haskell.org
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>
>>
>
>
> --
> /jve
>
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
Writing instances encode/decode that use either little endian or big endian
(or mixed!) is trivial... see attached file.
I don't see where your problem is...
On Thu, May 28, 2009 at 12:35 PM, David Leimbach wrote:
>
>
> On Thu, May 28, 2009 at 9:17 AM, John Van Enk wrote:
>
>>
>>> Fair enough. I am just new to the interface, wondering if I should
>>> try matching responses by pulling apart via Get, or the bit syntax
>>> package.
>>>
>>>
>>
>> I'm assming you have some 'data Foo = ...'? If this is the case, you're
>> probably okay writing an instance of Binary for Foo and using encode/decode.
>> That's all I've had to do and had superb performance.
>>
>>
>
> encode/decode do Big Endian, and 9P does little endian.
>
> From the man page:
>
> "Each message consists of a sequence of bytes. Two–, four–, and eight–byte
> fields hold unsigned integers represented in little–endian order (least
> significant byte first)."
>
> encode/decode just won't work for me as a result, as they assume the
> not-so-aptly-named (at least in this case) "network" byte order. Since I'm
> not Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you
> why they chose little endian for this.
>
> (I should note that I sometimes work with CAN hardware, and depending on
> what people want to put on the line, little endian creeps in there too, so
> this issue will have to be dealt with here as well, if I am to use
> Data.Binary)
>
> I've got (typing from memory, forgive syntax errors)
>
> module Nine (Message) where
>
> {-- missing imports --}
>
> notag = putWord16le -1
>
> -- Message type field values
> -- from: http://plan9.bell-labs.com/sources/plan9/sys/include/fcall.h
> tversion = 100
> rversion = 101
> rerror = 107
>
>
> -- only 9P2000 is being handled so some constructors are simple
> data Message = Tversion
> | Rversion {size :: Int, tag :: Int, msize :: Int,
> version :: String}
> | Rerror {message :: String} -- maybe this should
> be Maybe ByteString?
> ...
>
> instance Binary Message where
> put Tversion = putWord32le 16 >> tversion >> notag >> msize >> "9P2000"
> put Rversion = putWord32le 16 >> rversion >> notag >> msize >> "9P2000"
> get = do size <- getWord32le
> tag <- getWord8
> case tag of
> rversion -> do msize <- getWord32le
>version <- getRemainingLazyByteString
>return (Rversion size tag msize
> (C.unpack version))
>
> ===
>
> I'm beginning to feel the Get Monad is not going to be as nice as if I had
> some bit-syntax for determining what type of return message I'm getting.
> Any "T" message can be met with the paired "R" response, or an Rerror
> message, with a detailed description of what went wrong. My instance of get
> is looking like it's going to be pretty nasty :-)
>
> The first message in any 9p session is Tversion with Rversion or Rerror
> response. There's optional authentication hooks via Tauth/Rauth, and then
> the filesystem namespace navigation operation messages.
>
> Dave
>
> /jve
>>
>
>
> ___
> Haskell-Cafe mailing list
> Haskell-Cafe@haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
>
--
/jve
binaryTest.hs
Description: Binary data
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
leimy2k: > encode/decode do Big Endian, and 9P does little endian. > > From the man page: > > "Each message consists of a sequence of bytes. Two , four , and eight byte > fields hold unsigned integers represented in little endian order (least > significant byte first)." > > encode/decode just won't work for me as a result, as they assume the > not-so-aptly-named (at least in this case) "network" byte order. Since I'm > not > Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you why they > chose little endian for this. I think you misunderstand the API: encode and decode *use whatever the underlying instance for your data type uses*. If you write an instance that uses the little endian primitives, that is what I will use. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
On Thu, May 28, 2009 at 9:17 AM, John Van Enk wrote:
>
>> Fair enough. I am just new to the interface, wondering if I should
>> try matching responses by pulling apart via Get, or the bit syntax
>> package.
>>
>>
>
> I'm assming you have some 'data Foo = ...'? If this is the case, you're
> probably okay writing an instance of Binary for Foo and using encode/decode.
> That's all I've had to do and had superb performance.
>
>
encode/decode do Big Endian, and 9P does little endian.
>From the man page:
"Each message consists of a sequence of bytes. Two–, four–, and eight–byte
fields hold unsigned integers represented in little–endian order (least
significant byte first)."
encode/decode just won't work for me as a result, as they assume the
not-so-aptly-named (at least in this case) "network" byte order. Since I'm
not Rob Pike, or Ken Thompson or any of the Plan 9 guys, I can't tell you
why they chose little endian for this.
(I should note that I sometimes work with CAN hardware, and depending on
what people want to put on the line, little endian creeps in there too, so
this issue will have to be dealt with here as well, if I am to use
Data.Binary)
I've got (typing from memory, forgive syntax errors)
module Nine (Message) where
{-- missing imports --}
notag = putWord16le -1
-- Message type field values
-- from: http://plan9.bell-labs.com/sources/plan9/sys/include/fcall.h
tversion = 100
rversion = 101
rerror = 107
-- only 9P2000 is being handled so some constructors are simple
data Message = Tversion
| Rversion {size :: Int, tag :: Int, msize :: Int,
version :: String}
| Rerror {message :: String} -- maybe this should
be Maybe ByteString?
...
instance Binary Message where
put Tversion = putWord32le 16 >> tversion >> notag >> msize >> "9P2000"
put Rversion = putWord32le 16 >> rversion >> notag >> msize >> "9P2000"
get = do size <- getWord32le
tag <- getWord8
case tag of
rversion -> do msize <- getWord32le
version <- getRemainingLazyByteString
return (Rversion size tag msize (C.unpack
version))
===
I'm beginning to feel the Get Monad is not going to be as nice as if I had
some bit-syntax for determining what type of return message I'm getting.
Any "T" message can be met with the paired "R" response, or an Rerror
message, with a detailed description of what went wrong. My instance of get
is looking like it's going to be pretty nasty :-)
The first message in any 9p session is Tversion with Rversion or Rerror
response. There's optional authentication hooks via Tauth/Rauth, and then
the filesystem namespace navigation operation messages.
Dave
/jve
>
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
> > > Fair enough. I am just new to the interface, wondering if I should > try matching responses by pulling apart via Get, or the bit syntax > package. > > I'm assming you have some 'data Foo = ...'? If this is the case, you're probably okay writing an instance of Binary for Foo and using encode/decode. That's all I've had to do and had superb performance. /jve ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Data.Binary and little endian encoding
On Thursday, May 28, 2009, Don Stewart wrote: > leimy2k: >> I'm also trying to figure out how bad/good Haskell Binary IO really is that >> it's been addressed a few times differently :-) > > FWIW Binary IO as implemented in Data.Binary is widely used in our > production systems at Galois. I'd be fairly confident in it. > Fair enough. I am just new to the interface, wondering if I should try matching responses by pulling apart via Get, or the bit syntax package. > -- Don > ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Data.Binary and little endian encoding
This approach looks like it'd work just fine. On Fri, May 15, 2009 at 2:16 AM, John Lato wrote: > leimy2k: > > On Thu, May 14, 2009 at 8:46 PM, Don Stewart wrote: > > > >> leimy2k: > >> > > >> > > >> > On Thu, May 14, 2009 at 8:40 PM, Don Stewart wrote: > >> > > >> > leimy2k: > >> > > I actually need little endian encoding... wondering if anyone > else > >> hit > >> > this > >> > > with Data.Binary. (because I'm working with Bell Lab's 9P > protocol > >> which > >> > does > >> > > encode things on the network in little-endian order). > >> > > > >> > > Anyone got some "tricks" for this? > >> > > >> > Yes! > >> > There are big, little and host-endian primitives in the Get/Put > >> monads. > >> > > >> > > >> http://hackage.haskell.org/packages/archive/binary/0.5.0.1/doc/html/ > >> > Data-Binary-Put.html#v%3AputWord16le > >> > > >> > You can use these to build encoders directly. > >> > > >> > > >> > Cool... I just have to write my own encoder and decoder now. > >> > > >> > As a request could we get encodeLe decodeLe for a later version of > this > >> > library? :-) That'd be totally awesome. > >> > >> Oh, you mean entirely different instances for all the current ones, that > >> use LE encodings? > >> > > > > Well the library is leaning towards "Network Byte Order" in that it has > > encode/decode that only encode/decode for Big Endian. > > > > Us folks who have to do little endian all now have to write our own > > encoding/decoding :-) > > > > I'm speaking specifically of the encode/decode functions. I have no idea > > how they're implemented. > > The encode/decode functions just call the runGet/runPut functions for > whatever is being encoded or decoded to combined with the get/put > functions from the Binary instance. The endian-ness is entirely > determined by the Binary instance for the data type you're > encoding/decoding, not anything in the encode/decode functions > themselves. > > For data types you define, you can make the Binary instance LE > standard (although it may not be a good idea) by using the > little-endian primitives Don mentioned. For built-in types, if you > want to use a different endian-ness, you can make new get/put > functions then call runGet/runPut on those directly. You won't be > able to redefine the Binary instances, but if you'll be doing this > with a lot of different types you could make your own BinaryLE class > like this: > > class BinaryLE t where > putLE :: t -> Put > getLE :: Get t > > encodeLE :: BinaryLE a => a -> ByteString > encodeLE = runPut . putLE > > decodeLE :: BinaryLE a => ByteString -> a > decodeLE = runGet getLE > > I've done some work with Binary and little-endian, and this bias never > really bothered me (I think it's actually a host-endian bias, could be > mistaken). I was using my own data types so I had to make my own > custom Binary instances, which I would have had to do anyway. The > data format specified little-endian, so making the Binary instance > LE-standard seemed appropriate. > > You'd only need to re-make custom get/put functions if you're using > types that already have Binary instances, and you intend to manipulate > the data outside of Haskell. Does that describe your situation? > > John > ___ > Haskell-Cafe mailing list > Haskell-Cafe@haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe > ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Data.Binary and little endian encoding
leimy2k: > On Thu, May 14, 2009 at 8:46 PM, Don Stewart wrote: > >> leimy2k: >> > >> > >> > On Thu, May 14, 2009 at 8:40 PM, Don Stewart wrote: >> > >> > leimy2k: >> > > I actually need little endian encoding... wondering if anyone else >> hit >> > this >> > > with Data.Binary. (because I'm working with Bell Lab's 9P protocol >> which >> > does >> > > encode things on the network in little-endian order). >> > > >> > > Anyone got some "tricks" for this? >> > >> > Yes! >> > There are big, little and host-endian primitives in the Get/Put >> monads. >> > >> > >> http://hackage.haskell.org/packages/archive/binary/0.5.0.1/doc/html/ >> > Data-Binary-Put.html#v%3AputWord16le >> > >> > You can use these to build encoders directly. >> > >> > >> > Cool... I just have to write my own encoder and decoder now. >> > >> > As a request could we get encodeLe decodeLe for a later version of this >> > library? :-) That'd be totally awesome. >> >> Oh, you mean entirely different instances for all the current ones, that >> use LE encodings? >> > > Well the library is leaning towards "Network Byte Order" in that it has > encode/decode that only encode/decode for Big Endian. > > Us folks who have to do little endian all now have to write our own > encoding/decoding :-) > > I'm speaking specifically of the encode/decode functions. I have no idea > how they're implemented. The encode/decode functions just call the runGet/runPut functions for whatever is being encoded or decoded to combined with the get/put functions from the Binary instance. The endian-ness is entirely determined by the Binary instance for the data type you're encoding/decoding, not anything in the encode/decode functions themselves. For data types you define, you can make the Binary instance LE standard (although it may not be a good idea) by using the little-endian primitives Don mentioned. For built-in types, if you want to use a different endian-ness, you can make new get/put functions then call runGet/runPut on those directly. You won't be able to redefine the Binary instances, but if you'll be doing this with a lot of different types you could make your own BinaryLE class like this: class BinaryLE t where putLE :: t -> Put getLE :: Get t encodeLE :: BinaryLE a => a -> ByteString encodeLE = runPut . putLE decodeLE :: BinaryLE a => ByteString -> a decodeLE = runGet getLE I've done some work with Binary and little-endian, and this bias never really bothered me (I think it's actually a host-endian bias, could be mistaken). I was using my own data types so I had to make my own custom Binary instances, which I would have had to do anyway. The data format specified little-endian, so making the Binary instance LE-standard seemed appropriate. You'd only need to re-make custom get/put functions if you're using types that already have Binary instances, and you intend to manipulate the data outside of Haskell. Does that describe your situation? John ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
