Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
On Fri, 26 Jan 2007, Neil Davies wrote: > existing ecoding system - both the BER (Basic Encoding Rules) and the > PER (Packed Encoding Rules). If you are looking to target a well > supported standard - this would be the one. I'd say that ASN.1 encoding rules are badly, but widely supported. A surprisingly large number of security problems have been caused by ASN.1 code, and similar bugs have turned up in independent implementations so it isn't just one widespread shoddy implementation. OTOH ASN.1 is used by TLS, LDAP, Kerberos, SNMP, S/MIME, H.323, etc. etc. Tony. -- f.a.n.finch <[EMAIL PROTECTED]> http://dotat.at/ IRISH SEA: NORTHWEST 3 OR 4, OCCASIONALLY 5 AT FIRST. SLIGHT OR MODERATE. SHOWERS. GOOD. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
tomasz.zielonka: > On Fri, Jan 26, 2007 at 02:16:22PM +1100, Donald Bruce Stewart wrote: > > We believe so, and its a bug if this is not the case. > > > > The src documents the encoding format used for each type (we were unable > > to attach haddocks to instances.. grr.) > > > > All data is encoded in Network order, and extended to 64 bits for word > > sized values (like Int). It should be possible to encode a structure > > with ghc on x86, and decode it on a sparc64 running hugs. > > Did you consider using an encoding which uses variable number of bytes? > If yes, I would be interested to know your reason for not choosing such > an encoding. Efficiency? Yes, efficiency. If you look in tests/ there's a pretty heavy duty benchmark we use to compare against C. Sticking to word sized writes where possible is a big one (up to 10 fold). Interestingly, I did write an aligned-only, host-endian layer, and it was only some 10% faster on x86 over network order code. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
On Fri, Jan 26, 2007 at 04:36:50PM +0100, Tomasz Zielonka wrote: > Did you consider using an encoding which uses variable number of bytes? > If yes, I would be interested to know your reason for not choosing such > an encoding. Efficiency? I am testing/benchmarking one right now I wrote for 'Integer', so far, I think it may be better in time _and_ space! cache effects no doubt. A nice thing about it is that for the common case, short ascii strings, the serialized form takes up exactly as much as they would in C, very nice. :) John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
existing ecoding system - both the BER (Basic Encoding Rules) and the PER (Packed Encoding Rules). If you are looking to target a well supported standard - this would be the one. Neil On 26/01/07, Malcolm Wallace <[EMAIL PROTECTED]> wrote: Tomasz Zielonka <[EMAIL PROTECTED]> wrote: > Did you consider using an encoding which uses variable number of > bytes? If yes, I would be interested to know your reason for not > choosing such an encoding. Efficiency? My Binary implementation (from 1998) used a type-specific number of bits to encode the constructor - exactly as many as needed. (If you were writing custom instances, you could even use a variable number of bits for the constructor, e.g. using Huffman encoding to make the more common constructors have the shortest representation.) The latter certainly imposes an extra time overhead on decoding, because you cannot just take a fixed-size chunk of bits and have the value. But I would have thought that in the regular case, using a type-specific (but not constructor-specific) size for representing the constructor would be very easy and have no time overhead at all. Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
Tomasz Zielonka <[EMAIL PROTECTED]> wrote: > Did you consider using an encoding which uses variable number of > bytes? If yes, I would be interested to know your reason for not > choosing such an encoding. Efficiency? My Binary implementation (from 1998) used a type-specific number of bits to encode the constructor - exactly as many as needed. (If you were writing custom instances, you could even use a variable number of bits for the constructor, e.g. using Huffman encoding to make the more common constructors have the shortest representation.) The latter certainly imposes an extra time overhead on decoding, because you cannot just take a fixed-size chunk of bits and have the value. But I would have thought that in the regular case, using a type-specific (but not constructor-specific) size for representing the constructor would be very easy and have no time overhead at all. Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
On Fri, Jan 26, 2007 at 02:16:22PM +1100, Donald Bruce Stewart wrote: > We believe so, and its a bug if this is not the case. > > The src documents the encoding format used for each type (we were unable > to attach haddocks to instances.. grr.) > > All data is encoded in Network order, and extended to 64 bits for word > sized values (like Int). It should be possible to encode a structure > with ghc on x86, and decode it on a sparc64 running hugs. Did you consider using an encoding which uses variable number of bytes? If yes, I would be interested to know your reason for not choosing such an encoding. Efficiency? Best regards Tomasz ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
john: > On Thu, Jan 25, 2007 at 07:11:55PM -0800, John Meacham wrote: > > Is the binary format portable? I need the produced files to work on both > > 32 and 64 bit architectures and with big and little endian machines. And > > of course, between different versions of a compiler or different > > compilers. > > Sorry to reply to myself, looking at the code, I see that it is. > however, Ints appear to be stored as 64 bits always, this seems like a > mistake. The Haskell standard only specifies Ints must have at least 30 > bits of precision so programs that rely on more than that are not > portable anyway. Plus, it is unlikely that any compilers ever will have > Ints > 32 bits, ghc does at the moment by accident of design and it is > considered a misfeature that will be fixed at some point. It would be an > ugly wart to be stuck with going forward... This was perhaps the only issue of contention during development. It was felt that those wishing to serialise precisely would use an explicit word sized type, such as Int64 or Word32, and that having things work correctly on ghc/amd64 right now was critical. If the Int/Int64 issue is resolved in the future, we can revisit this. Its fairly painless to upgrade files from one version of a Binary instance to another too (you just read in using the old 'get' method, and write back out using the new 'put' method). -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
On Thu, Jan 25, 2007 at 07:11:55PM -0800, John Meacham wrote: > Is the binary format portable? I need the produced files to work on both > 32 and 64 bit architectures and with big and little endian machines. And > of course, between different versions of a compiler or different > compilers. Sorry to reply to myself, looking at the code, I see that it is. however, Ints appear to be stored as 64 bits always, this seems like a mistake. The Haskell standard only specifies Ints must have at least 30 bits of precision so programs that rely on more than that are not portable anyway. Plus, it is unlikely that any compilers ever will have Ints > 32 bits, ghc does at the moment by accident of design and it is considered a misfeature that will be fixed at some point. It would be an ugly wart to be stuck with going forward... John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
john: > Yay! I knew if I waited long enough someone would write this. > > Is the binary format portable? I need the produced files to work on both > 32 and 64 bit architectures and with big and little endian machines. And > of course, between different versions of a compiler or different > compilers. We believe so, and its a bug if this is not the case. The src documents the encoding format used for each type (we were unable to attach haddocks to instances.. grr.) All data is encoded in Network order, and extended to 64 bits for word sized values (like Int). It should be possible to encode a structure with ghc on x86, and decode it on a sparc64 running hugs. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
Yay! I knew if I waited long enough someone would write this. Is the binary format portable? I need the produced files to work on both 32 and 64 bit architectures and with big and little endian machines. And of course, between different versions of a compiler or different compilers. John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: [Haskell] ANNOUNCE: binary: high performance, pure binary serialisation
dons: > > Binary: high performance, pure binary serialisation for Haskell > -- > > The Binary Strike Team is pleased to announce the release of a new, > pure, efficient binary serialisation library for Haskell, now available > from Hackage: Ok, I forgot one point. It is possible to automatically derive instances of Binary for your custom types, if they inhabit Data and Typeable, using an SYB trick. Load tools/derive/BinaryDerive.hs into ghci, and bring your type into scope, then run: *Main> mapM_ putStrLn . lines $ derive (undefined :: Drinks) To have the source for the Binary instance for the type Drinks derivied for you: *Main> mapM_ putStrLn . lines $ derive (undefined :: Drinks) instance Binary Main.Drinks where put (Beer a) = putWord8 0 >> put a put Coffee = putWord8 1 put Tea = putWord8 2 put EnergyDrink = putWord8 3 put Water = putWord8 4 put Wine = putWord8 5 put Whisky = putWord8 6 get = do tag_ <- getWord8 case tag_ of 0 -> get >>= \a -> return (Beer a) 1 -> return Coffee 2 -> return Tea 3 -> return EnergyDrink 4 -> return Water 5 -> return Wine 6 -> return Whisky The use of SYB techniques to provide a 'deriving' script along with a new typeclass seems to be quite handy. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
