Re: ANN: Gloss, a byte-format DSL
It is not. Thanks for the information anyway. The real problem consists of a set of tags, with a type identified by a byte. One of those tags is a compound tag which can contain any number of other tags and is terminated by a \0. This means I need parser behavior, rather than a linear regex-alike behavior. I would totally understand if Gloss does not, and will never provide this type of behavior. If it did, that'd mean you could write an XML parser in Gloss. :) Thanks for your help. While Gloss might not solve this particular problem, I'm sure I'll find lots of other problems to use it for. Pepijn On Jan 10, 11:01 pm, Zach Tellman ztell...@gmail.com wrote: I don't know if your example codec is as simple as your real problem, but here's a codec that will work for the string you provided: (repeated (string :utf-8 :delimiters [\n \n\0]) :delimiters [\n\0] :strip-delimiters? false) This terminates the whole sequence only on \n\0, but doesn't strip out the terminator so that it can be used by the last string as well. Right now this will not properly encode (the last token will be \n\n \0, so there will be a spurious empty string at the end of the list), but I plan on fixing that soon. Zach On Jan 10, 11:47 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Oh, right. My code does have a start and an end. I'm using header for the start. So the only way Gloss could do this is with a fn that combines header and repeated into one? On Jan 10, 12:29 pm, Ken Wesson kwess...@gmail.com wrote: On Mon, Jan 10, 2011 at 4:52 AM, pepijn (aka fliebel) pepijnde...@gmail.com wrote: The later. The string in my example is \n terminated, so it should read past the \0, and then the outer list should terminate on the last \0. My point is that I need to parse recursive structures, which of course contain the same terminator as the outer one. What if I wanted to parse null terminated lists of null terminated lists of ... of bytes? Like so: abc\0def\0\0 def\0ghi\0jkl\0\0 \0 Your implementation would just make the outer list read to the first \0 and call it a day. What I need is that it only checks for a terminator on the outer list after reading a full inner list, and keep doing that, until it finds a terminator right after the inner list. That's not going to work unless all leaves are at the same depth and there are no empty lists at any depth. If the leaves are all at the same depth: (defn unflatten ([in-seq sentinel] (unflatten (seq in-seq) sentinel [[]] 0)) ([in-seq sentinel stack height] (if in-seq (let [[f r] in-seq] (if (= f sentinel) (let [h (inc height) l1 (get stack h) l1 (if l1 l1 []) l2 (get stack height) s1 (assoc stack h (conj l1 l2)) s2 (assoc s1 height [])] (recur r sentinel s2 (inc height))) (let [l1 (get stack 0) s1 (assoc stack 0 (conj l1 f))] (recur r sentinel s1 0 (peek stack user= (unflatten [1 2 0 3 4 0 0 5 6 0 7 8 0 0] 0) [[[1 2] [3 4]] [[5 6] [7 8]]] Otherwise you're going to need to have a start-of-list delimiter as well as an end-of-list delimiter: (defn unflatten ([in-seq start-del end-del] (unflatten (seq in-seq) start-del end-del (list []))) ([in-seq s e stack] (if in-seq (let [[f r] in-seq] (condp = f e (let [[x y z] stack] (if y (recur r s e (conj z (conj y x))) (throw (Error. s (recur r s e (conj stack [])) (let [[x y] stack] (recur r s e (conj y (conj x f)) (if ( (count stack) 1) (throw (Error.)) (first stack) user= (unflatten (((ab)(cd))(ef)(g(hi))) \( \)) \a \b] [\c \d]] [\e \f] [\g [\h \i -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
The later. The string in my example is \n terminated, so it should read past the \0, and then the outer list should terminate on the last \0. My point is that I need to parse recursive structures, which of course contain the same terminator as the outer one. What if I wanted to parse null terminated lists of null terminated lists of ... of bytes? Like so: abc\0def\0\0 def\0ghi\0jkl\0\0 \0 Your implementation would just make the outer list read to the first \0 and call it a day. What I need is that it only checks for a terminator on the outer list after reading a full inner list, and keep doing that, until it finds a terminator right after the inner list. Pepijn On Jan 6, 7:17 am, Zach Tellman ztell...@gmail.com wrote: I'm confused by what you expect the decoded value for that string to look like. Is it: [[blabla hi hi jg] [ g]] or [[blabla hi hi jg\0 g]] Zach On Jan 5, 3:36 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Nothing :( $ lein repl REPL started; server listening on localhost:32399. user= (use '[gloss core io]) nil user= (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) #'user/t user= (decode t (.getBytes blabla\nhihi\njg\0 g\n\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. (NO_SOURCE_FILE:0) What I think is happening is that repeated reads up to the first \0 and then tries to fit the subframes inside of that. What I think it *should* do is, check the next byte for a delimiter, if not, read a subframe, rinse and repeat. Pepijn On Jan 2, 7:38 pm, Zach Tellman ztell...@gmail.com wrote: There was a bug in repeated, which is fixed now. Pull the latest from github or clojars and let me know how it goes. Zach On Jan 2, 3:29 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Okay, clicked send to early. The header also contains 0-bytes, so the repeated stops 'mid-sentence' and tries to balance things. Is there any wayGlosscan handle nested structures like this? On Jan 2, 12:20 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point inGlossso far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am tryingGlossfor reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly
Re: ANN: Gloss, a byte-format DSL
On Mon, Jan 10, 2011 at 4:52 AM, pepijn (aka fliebel) pepijnde...@gmail.com wrote: The later. The string in my example is \n terminated, so it should read past the \0, and then the outer list should terminate on the last \0. My point is that I need to parse recursive structures, which of course contain the same terminator as the outer one. What if I wanted to parse null terminated lists of null terminated lists of ... of bytes? Like so: abc\0def\0\0 def\0ghi\0jkl\0\0 \0 Your implementation would just make the outer list read to the first \0 and call it a day. What I need is that it only checks for a terminator on the outer list after reading a full inner list, and keep doing that, until it finds a terminator right after the inner list. That's not going to work unless all leaves are at the same depth and there are no empty lists at any depth. If the leaves are all at the same depth: (defn unflatten ([in-seq sentinel] (unflatten (seq in-seq) sentinel [[]] 0)) ([in-seq sentinel stack height] (if in-seq (let [[f r] in-seq] (if (= f sentinel) (let [h (inc height) l1 (get stack h) l1 (if l1 l1 []) l2 (get stack height) s1 (assoc stack h (conj l1 l2)) s2 (assoc s1 height [])] (recur r sentinel s2 (inc height))) (let [l1 (get stack 0) s1 (assoc stack 0 (conj l1 f))] (recur r sentinel s1 0 (peek stack user= (unflatten [1 2 0 3 4 0 0 5 6 0 7 8 0 0] 0) [[[1 2] [3 4]] [[5 6] [7 8]]] Otherwise you're going to need to have a start-of-list delimiter as well as an end-of-list delimiter: (defn unflatten ([in-seq start-del end-del] (unflatten (seq in-seq) start-del end-del (list []))) ([in-seq s e stack] (if in-seq (let [[f r] in-seq] (condp = f e (let [[x y z] stack] (if y (recur r s e (conj z (conj y x))) (throw (Error. s (recur r s e (conj stack [])) (let [[x y] stack] (recur r s e (conj y (conj x f)) (if ( (count stack) 1) (throw (Error.)) (first stack) user= (unflatten (((ab)(cd))(ef)(g(hi))) \( \)) \a \b] [\c \d]] [\e \f] [\g [\h \i -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Oh, right. My code does have a start and an end. I'm using header for the start. So the only way Gloss could do this is with a fn that combines header and repeated into one? On Jan 10, 12:29 pm, Ken Wesson kwess...@gmail.com wrote: On Mon, Jan 10, 2011 at 4:52 AM, pepijn (aka fliebel) pepijnde...@gmail.com wrote: The later. The string in my example is \n terminated, so it should read past the \0, and then the outer list should terminate on the last \0. My point is that I need to parse recursive structures, which of course contain the same terminator as the outer one. What if I wanted to parse null terminated lists of null terminated lists of ... of bytes? Like so: abc\0def\0\0 def\0ghi\0jkl\0\0 \0 Your implementation would just make the outer list read to the first \0 and call it a day. What I need is that it only checks for a terminator on the outer list after reading a full inner list, and keep doing that, until it finds a terminator right after the inner list. That's not going to work unless all leaves are at the same depth and there are no empty lists at any depth. If the leaves are all at the same depth: (defn unflatten ([in-seq sentinel] (unflatten (seq in-seq) sentinel [[]] 0)) ([in-seq sentinel stack height] (if in-seq (let [[f r] in-seq] (if (= f sentinel) (let [h (inc height) l1 (get stack h) l1 (if l1 l1 []) l2 (get stack height) s1 (assoc stack h (conj l1 l2)) s2 (assoc s1 height [])] (recur r sentinel s2 (inc height))) (let [l1 (get stack 0) s1 (assoc stack 0 (conj l1 f))] (recur r sentinel s1 0 (peek stack user= (unflatten [1 2 0 3 4 0 0 5 6 0 7 8 0 0] 0) [[[1 2] [3 4]] [[5 6] [7 8]]] Otherwise you're going to need to have a start-of-list delimiter as well as an end-of-list delimiter: (defn unflatten ([in-seq start-del end-del] (unflatten (seq in-seq) start-del end-del (list []))) ([in-seq s e stack] (if in-seq (let [[f r] in-seq] (condp = f e (let [[x y z] stack] (if y (recur r s e (conj z (conj y x))) (throw (Error. s (recur r s e (conj stack [])) (let [[x y] stack] (recur r s e (conj y (conj x f)) (if ( (count stack) 1) (throw (Error.)) (first stack) user= (unflatten (((ab)(cd))(ef)(g(hi))) \( \)) \a \b] [\c \d]] [\e \f] [\g [\h \i -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
I don't know if your example codec is as simple as your real problem, but here's a codec that will work for the string you provided: (repeated (string :utf-8 :delimiters [\n \n\0]) :delimiters [\n\0] :strip-delimiters? false) This terminates the whole sequence only on \n\0, but doesn't strip out the terminator so that it can be used by the last string as well. Right now this will not properly encode (the last token will be \n\n \0, so there will be a spurious empty string at the end of the list), but I plan on fixing that soon. Zach On Jan 10, 11:47 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Oh, right. My code does have a start and an end. I'm using header for the start. So the only way Gloss could do this is with a fn that combines header and repeated into one? On Jan 10, 12:29 pm, Ken Wesson kwess...@gmail.com wrote: On Mon, Jan 10, 2011 at 4:52 AM, pepijn (aka fliebel) pepijnde...@gmail.com wrote: The later. The string in my example is \n terminated, so it should read past the \0, and then the outer list should terminate on the last \0. My point is that I need to parse recursive structures, which of course contain the same terminator as the outer one. What if I wanted to parse null terminated lists of null terminated lists of ... of bytes? Like so: abc\0def\0\0 def\0ghi\0jkl\0\0 \0 Your implementation would just make the outer list read to the first \0 and call it a day. What I need is that it only checks for a terminator on the outer list after reading a full inner list, and keep doing that, until it finds a terminator right after the inner list. That's not going to work unless all leaves are at the same depth and there are no empty lists at any depth. If the leaves are all at the same depth: (defn unflatten ([in-seq sentinel] (unflatten (seq in-seq) sentinel [[]] 0)) ([in-seq sentinel stack height] (if in-seq (let [[f r] in-seq] (if (= f sentinel) (let [h (inc height) l1 (get stack h) l1 (if l1 l1 []) l2 (get stack height) s1 (assoc stack h (conj l1 l2)) s2 (assoc s1 height [])] (recur r sentinel s2 (inc height))) (let [l1 (get stack 0) s1 (assoc stack 0 (conj l1 f))] (recur r sentinel s1 0 (peek stack user= (unflatten [1 2 0 3 4 0 0 5 6 0 7 8 0 0] 0) [[[1 2] [3 4]] [[5 6] [7 8]]] Otherwise you're going to need to have a start-of-list delimiter as well as an end-of-list delimiter: (defn unflatten ([in-seq start-del end-del] (unflatten (seq in-seq) start-del end-del (list []))) ([in-seq s e stack] (if in-seq (let [[f r] in-seq] (condp = f e (let [[x y z] stack] (if y (recur r s e (conj z (conj y x))) (throw (Error. s (recur r s e (conj stack [])) (let [[x y] stack] (recur r s e (conj y (conj x f)) (if ( (count stack) 1) (throw (Error.)) (first stack) user= (unflatten (((ab)(cd))(ef)(g(hi))) \( \)) \a \b] [\c \d]] [\e \f] [\g [\h \i -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Hey Eric
I did something similar for handling unsigned types. I think it's
simpler and probably faster, not that I've checked. Here's a gist of
it https://gist.github.com/767887
That code is part of a little library I wrote a while ago
https://github.com/geoffsalmon/bytebuffer It's closer to your bin-ed
in complexity than to gloss. You might also find the pack-bits and
unpack-bits functions useful if you're parsing bit-fields and not
using Gloss's bits-seq.
I'm going to give Gloss a try. For complex binary structures, I think
I like it's way of specifying formats using (keyword, type) pairs and
reading and writing maps instead of bytebuffer's which specifies
formats as strings ('b' for byte, 'i' for int, and so on) and reads
and writes seqs of values.
- Geoff
On Jan 5, 5:45 pm, Eric Schulte schulte.e...@gmail.com wrote:
also, here's a patch to Gloss which I've used locally but which may be
generally useful.
Cheers -- Eric
0001-adding-primitive-support-for-uint16-uint32-and-uint6.patch
1KViewDownload
Eric Schulte schulte.e...@gmail.com writes:
Thanks for sharing this library, I found reading your code to be very
useful.
I need to be able to read and write elf files from within Clojure code,
and Gloss initially looked like a good option. However much of the
streaming support and frame-based conception got in the way of my
particular needs.
In the end I have written a much simpler collection of functions which
supports parsing binary files into lists of bytes, manipulating these
lists, and then writing the lists back to binary files. This is heavily
influenced by the code in Gloss. Seehttps://github.com/eschulte/bin-ed
Anyways just thought I'd share, if anyone notices anything in the code
which doesn't make sense, or could be more easily accomplished in some
other way please do let me know.
Thanks -- Eric
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Hey Geoff,
This sort of feedback and code recommendations is exactly what I was
hoping for.
Geoff geoff.sal...@gmail.com writes:
Hey Eric
I did something similar for handling unsigned types. I think it's
simpler and probably faster, not that I've checked. Here's a gist of
it https://gist.github.com/767887
Ah, these do look like an improvement, thanks for sharing.
That code is part of a little library I wrote a while ago
https://github.com/geoffsalmon/bytebuffer It's closer to your bin-ed
in complexity than to gloss. You might also find the pack-bits and
unpack-bits functions useful if you're parsing bit-fields and not
using Gloss's bits-seq.
Very nice, I'm actually using vectors to fake ordered maps to mimic
Gloss' (keyword value) codecs, e.g. https://gist.github.com/768101
Cheers -- Eric
I'm going to give Gloss a try. For complex binary structures, I think
I like it's way of specifying formats using (keyword, type) pairs and
reading and writing maps instead of bytebuffer's which specifies
formats as strings ('b' for byte, 'i' for int, and so on) and reads
and writes seqs of values.
- Geoff
On Jan 5, 5:45 pm, Eric Schulte schulte.e...@gmail.com wrote:
also, here's a patch to Gloss which I've used locally but which may be
generally useful.
Cheers -- Eric
0001-adding-primitive-support-for-uint16-uint32-and-uint6.patch
1KViewDownload
Eric Schulte schulte.e...@gmail.com writes:
Thanks for sharing this library, I found reading your code to be very
useful.
I need to be able to read and write elf files from within Clojure code,
and Gloss initially looked like a good option. However much of the
streaming support and frame-based conception got in the way of my
particular needs.
In the end I have written a much simpler collection of functions which
supports parsing binary files into lists of bytes, manipulating these
lists, and then writing the lists back to binary files. This is heavily
influenced by the code in Gloss. Seehttps://github.com/eschulte/bin-ed
Anyways just thought I'd share, if anyone notices anything in the code
which doesn't make sense, or could be more easily accomplished in some
other way please do let me know.
Thanks -- Eric
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Nothing :( $ lein repl REPL started; server listening on localhost:32399. user= (use '[gloss core io]) nil user= (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) #'user/t user= (decode t (.getBytes blabla\nhihi\njg\0 g\n\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. (NO_SOURCE_FILE:0) What I think is happening is that repeated reads up to the first \0 and then tries to fit the subframes inside of that. What I think it *should* do is, check the next byte for a delimiter, if not, read a subframe, rinse and repeat. Pepijn On Jan 2, 7:38 pm, Zach Tellman ztell...@gmail.com wrote: There was a bug in repeated, which is fixed now. Pull the latest from github or clojars and let me know how it goes. Zach On Jan 2, 3:29 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Okay, clicked send to early. The header also contains 0-bytes, so the repeated stops 'mid-sentence' and tries to balance things. Is there any way Gloss can handle nested structures like this? On Jan 2, 12:20 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point in Gloss so far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am trying Gloss for reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian ...) and (big-endian ...), with big-endian as the default. Just as a side-note, though, Calx [1] is already handling little- endian data by using encode-to-buffer, where it's writing to a buffer whose endianness has been preset. This obviously isn't a general solution, but just thought
Re: ANN: Gloss, a byte-format DSL
Thanks for sharing this library, I found reading your code to be very useful. I need to be able to read and write elf files from within Clojure code, and Gloss initially looked like a good option. However much of the streaming support and frame-based conception got in the way of my particular needs. In the end I have written a much simpler collection of functions which supports parsing binary files into lists of bytes, manipulating these lists, and then writing the lists back to binary files. This is heavily influenced by the code in Gloss. See https://github.com/eschulte/bin-ed Anyways just thought I'd share, if anyone notices anything in the code which doesn't make sense, or could be more easily accomplished in some other way please do let me know. Thanks -- Eric -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
also, here's a patch to Gloss which I've used locally but which may be
generally useful.
Cheers -- Eric
Eric Schulte schulte.e...@gmail.com writes:
Thanks for sharing this library, I found reading your code to be very
useful.
I need to be able to read and write elf files from within Clojure code,
and Gloss initially looked like a good option. However much of the
streaming support and frame-based conception got in the way of my
particular needs.
In the end I have written a much simpler collection of functions which
supports parsing binary files into lists of bytes, manipulating these
lists, and then writing the lists back to binary files. This is heavily
influenced by the code in Gloss. See https://github.com/eschulte/bin-ed
Anyways just thought I'd share, if anyone notices anything in the code
which doesn't make sense, or could be more easily accomplished in some
other way please do let me know.
Thanks -- Eric
From a5888053767a0a3de798e432a15b26d98fa8b78f Mon Sep 17 00:00:00 2001
From: Eric Schulte schulte.e...@gmail.com
Date: Mon, 3 Jan 2011 19:33:07 -0500
Subject: [PATCH] adding primitive support for uint16, uint32 and uint64
---
src/gloss/data/primitives.clj | 15 +++
1 files changed, 15 insertions(+), 0 deletions(-)
diff --git a/src/gloss/data/primitives.clj b/src/gloss/data/primitives.clj
index 6f8ad4c..f524d21 100644
--- a/src/gloss/data/primitives.clj
+++ b/src/gloss/data/primitives.clj
@@ -26,6 +26,18 @@
(string? x) (- x first int byte)
:else (throw (Exception. (str Cannot convert x to byte.)
+(defn uint16 Convert to a uint16 [x]
+ (let [int16 (short x)]
+(if (neg? int16) (+ 0x1 int16) int16)))
+
+(defn uint32 Convert to a uint32 [x]
+ (let [int32 (int x)]
+(if (neg? int32) (+ 0x1 int32) int32)))
+
+(defn uint64 Convert to a uint64 [x]
+ (let [int64 (long x)]
+(if (neg? int64) (+ 0x1 int64) int64)))
+
(defmacro primitive-codec [accessor writer size typecast transform-fn]
`(reify
Reader
@@ -54,7 +66,10 @@
(def primitive-codecs
{:byte (primitive-codec .get .put 1 byte to-byte)
:int16 (primitive-codec .getShort .putShort 2 short identity)
+ :uint16 (primitive-codec .getShort .putShort 2 uint16 identity)
:int32 (primitive-codec .getInt .putInt 4 int identity)
+ :uint32 (primitive-codec .getInt .putInt 4 uint32 identity)
:int64 (primitive-codec .getLong .putLong 8 long identity)
+ :uint64 (primitive-codec .getLong .putLong 8 uint64 identity)
:float32 (primitive-codec .getFloat .putFloat 4 float identity)
:float64 (primitive-codec .getDouble .putDouble 8 double identity)})
--
1.7.1
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
I'm confused by what you expect the decoded value for that string to look like. Is it: [[blabla hi hi jg] [ g]] or [[blabla hi hi jg\0 g]] Zach On Jan 5, 3:36 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Nothing :( $ lein repl REPL started; server listening on localhost:32399. user= (use '[gloss core io]) nil user= (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) #'user/t user= (decode t (.getBytes blabla\nhihi\njg\0 g\n\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. (NO_SOURCE_FILE:0) What I think is happening is that repeated reads up to the first \0 and then tries to fit the subframes inside of that. What I think it *should* do is, check the next byte for a delimiter, if not, read a subframe, rinse and repeat. Pepijn On Jan 2, 7:38 pm, Zach Tellman ztell...@gmail.com wrote: There was a bug in repeated, which is fixed now. Pull the latest from github or clojars and let me know how it goes. Zach On Jan 2, 3:29 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Okay, clicked send to early. The header also contains 0-bytes, so the repeated stops 'mid-sentence' and tries to balance things. Is there any wayGlosscan handle nested structures like this? On Jan 2, 12:20 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point inGlossso far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am tryingGlossfor reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian
Re: ANN: Gloss, a byte-format DSL
Thanks for the patch! It's too bad that Gloss wasn't directly suited to your needs, but I appreciate you taking the time to familiarize yourself with the code and add new functionality. Zach On Jan 5, 2:45 pm, Eric Schulte schulte.e...@gmail.com wrote: also, here's a patch to Gloss which I've used locally but which may be generally useful. Cheers -- Eric 0001-adding-primitive-support-for-uint16-uint32-and-uint6.patch 1KViewDownload Eric Schulte schulte.e...@gmail.com writes: Thanks for sharing this library, I found reading your code to be very useful. I need to be able to read and write elf files from within Clojure code, and Gloss initially looked like a good option. However much of the streaming support and frame-based conception got in the way of my particular needs. In the end I have written a much simpler collection of functions which supports parsing binary files into lists of bytes, manipulating these lists, and then writing the lists back to binary files. This is heavily influenced by the code in Gloss. Seehttps://github.com/eschulte/bin-ed Anyways just thought I'd share, if anyone notices anything in the code which doesn't make sense, or could be more easily accomplished in some other way please do let me know. Thanks -- Eric -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point in Gloss so far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am trying Gloss for reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian ...) and (big-endian ...), with big-endian as the default. Just as a side-note, though, Calx [1] is already handling little- endian data by using encode-to-buffer, where it's writing to a buffer whose endianness has been preset. This obviously isn't a general solution, but just thought I'd point it out. Zach [1]https://github.com/ztellman/calx On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote: If Gloss is to decode incoming packet (byte array) in little-endian format it is straightforward: Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and pass b to decode function that will return Clojure map of decoded values. However, when outgoing packet byte array is to be produced from map of values, encode function will always return ByteBuffer in default big- endian format, so resulting byte array extracted form ByteBuffer using get() method will be incorrect. If Gloss is to support little-endian frames, it seems that endianness needs to be part of frame definition. In that case Gloss decode fun would refuse to accept ByteBuffers with wrong order() and encode fun will always generate the correct result. Zoka On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote: ByteBuffers have an order() method which allows you to toggle the endianness. I haven't tested this, but since everything is built on top of Java's ByteBuffer functionality it should be fine as long as the ByteBuffers are correctly set and correctly ordered with respect to each other. Zach On Nov 23, 2:52 pm, zoka
Re: ANN: Gloss, a byte-format DSL
Okay, clicked send to early. The header also contains 0-bytes, so the repeated stops 'mid-sentence' and tries to balance things. Is there any way Gloss can handle nested structures like this? On Jan 2, 12:20 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point in Gloss so far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am trying Gloss for reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian ...) and (big-endian ...), with big-endian as the default. Just as a side-note, though, Calx [1] is already handling little- endian data by using encode-to-buffer, where it's writing to a buffer whose endianness has been preset. This obviously isn't a general solution, but just thought I'd point it out. Zach [1]https://github.com/ztellman/calx On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote: If Gloss is to decode incoming packet (byte array) in little-endian format it is straightforward: Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and pass b to decode function that will return Clojure map of decoded values. However, when outgoing packet byte array is to be produced from map of values, encode function will always return ByteBuffer in default big- endian format, so resulting byte array extracted form ByteBuffer using get() method will be incorrect. If Gloss is to support little-endian frames, it seems that endianness needs to be part of frame definition. In that case Gloss decode fun would refuse to accept ByteBuffers with wrong order() and encode fun will always generate the correct result. Zoka On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote:
Re: ANN: Gloss, a byte-format DSL
There was a bug in repeated, which is fixed now. Pull the latest from github or clojars and let me know how it goes. Zach On Jan 2, 3:29 am, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Okay, clicked send to early. The header also contains 0-bytes, so the repeated stops 'mid-sentence' and tries to balance things. Is there any way Gloss can handle nested structures like this? On Jan 2, 12:20 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hi, Thanks for helping out. After using codecs rather than frames, I get even weirder errors. My code now gives me: java.lang.Exception: Cannot evenly divide bytes into sequence of frames. Hard coding the header followed by a terminating :byte works as it should: (decode (compile-frame [tag tstring (header tag (memoize #(compile- frame [(name %) tstring (get-tag %)])) (comp symbol first)) :byte]) data) [:compound hello world [string name Bananrama] 0] So the problem seems to be with the repeated. Investigating that, I copied an example from the introduction page, and this is the result: (defcodec t (repeated (string :utf-8 :delimiters [\n]) :delimiters [\0])) (encode t [foo bar baz]) java.lang.IllegalArgumentException: Don't know how to create ISeq from: java.nio.HeapByteBuffer ((#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) (#HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=3 cap=3] #HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]) But on the other hand: (decode t (.getBytes blabla\nhihi\ng\n\0)) [blabla hihi g] This gives me the same error as my code, but since the header in my code seems correct, I don't see why it has leftover bytes. (decode t (.getBytes blabla\nhihi\ng\0)) java.lang.Exception: Cannot evenly divide bytes into sequence of frames. By the way, is there an easy way to get something readable out of encode? Like Unix hexdump, or even just a seq of integers. Debugging is a weak point in Gloss so far, if you ask me. Thanks! Pepijn de Vos On Jan 1, 10:47 pm, Zach Tellman ztell...@gmail.com wrote: The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found athttps://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am trying Gloss for reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian ...) and (big-endian ...), with big-endian as the default. Just as a side-note, though, Calx [1] is already handling little- endian data by using encode-to-buffer, where it's writing to a buffer whose endianness has been preset. This obviously isn't a general solution, but just thought I'd point it out. Zach [1]https://github.com/ztellman/calx On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote: If Gloss is to decode incoming packet (byte array) in little-endian format it is straightforward: Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and pass b to decode function that will return Clojure map of decoded values. However, when outgoing packet byte array is to be produced from map of values, encode function will always return ByteBuffer in default big- endian format, so resulting byte array extracted form ByteBuffer using get() method will be incorrect. If Gloss is to support little-endian frames, it
Re: ANN: Gloss, a byte-format DSL
Hey,
I am trying Gloss for reading NBT [1] files.
First thing I did like is that it seems to make things real easy.
First thing I did not like is the weak separation between types
like :byte and extra data like :foo.
I think I'm nearly done with the NBT reader [2], but I ran into a
problem. Whatever I put in the header form, I get exceptions like
this:
java.lang.IllegalArgumentException: No implementation of
method: :sizeof of protocol: #'gloss.core.protocols/Writer found for
class: clojure.lang.PersistentVector
Only thing it mentions in the stacktrace [3] is methods on a reify,
which calls the same method again, or in the most recent case, just
return nil.
[1] http://www.minecraft.net/docs/NBT.txt
[2] https://gist.github.com/761997
[3] http://pastebin.com/AqrsbjuS
On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote:
You're right, that's an omission from the frame syntax. I'll add the
ability for all or part of the frame to be scoped as (little-
endian ...) and (big-endian ...), with big-endian as the default.
Just as a side-note, though, Calx [1] is already handling little-
endian data by using encode-to-buffer, where it's writing to a buffer
whose endianness has been preset. This obviously isn't a general
solution, but just thought I'd point it out.
Zach
[1]https://github.com/ztellman/calx
On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote:
If Gloss is to decode incoming packet (byte array) in little-endian
format it is straightforward:
Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and
pass b to decode function that will return Clojure map of decoded
values.
However, when outgoing packet byte array is to be produced from map of
values, encode function will always return ByteBuffer in default big-
endian format, so resulting byte array extracted form ByteBuffer using
get() method will be incorrect.
If Gloss is to support little-endian frames, it seems that endianness
needs to be part of frame definition. In that case Gloss decode fun
would refuse to accept ByteBuffers with wrong order() and encode fun
will always generate the correct result.
Zoka
On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote:
ByteBuffers have an order() method which allows you to toggle the
endianness. I haven't tested this, but since everything is built on
top of Java's ByteBuffer functionality it should be fine as long as
the ByteBuffers are correctly set and correctly ordered with respect
to each other.
Zach
On Nov 23, 2:52 pm, zoka ztomi...@gmail.com wrote:
JVM stores numbers in in big endian format - is there a way to process
binary stream containing little endian numbers?
Zoka
On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal
with
mixed C datatypes in Java. When writing Aleph [2], I discovered
the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully
help a
few other people out, I've written Gloss, which can transform a
simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found
athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a
fixed
format that cannot be used to interface with external systems.
Gloss
is less performant than protocol buffers, but is also much less
picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How
Re: ANN: Gloss, a byte-format DSL
The header-body function in (header ...) must return a codec, so you need to call compile-frame on the vector you're generating. Since you don't want to call compile-frame every time you decode a frame, you can memoize the function. A version that does both can be found at https://gist.github.com/762031. I agree that the way the enumeration and types are blurred in your code is a little confusing. You could create a stronger distinction by calling your enumerated types :tag-byte, :tag-int32, etc, and then defining a map from those tags onto :byte, :int32, and so on. Zach On Jan 1, 1:01 pm, pepijn (aka fliebel) pepijnde...@gmail.com wrote: Hey, I am trying Gloss for reading NBT [1] files. First thing I did like is that it seems to make things real easy. First thing I did not like is the weak separation between types like :byte and extra data like :foo. I think I'm nearly done with the NBT reader [2], but I ran into a problem. Whatever I put in the header form, I get exceptions like this: java.lang.IllegalArgumentException: No implementation of method: :sizeof of protocol: #'gloss.core.protocols/Writer found for class: clojure.lang.PersistentVector Only thing it mentions in the stacktrace [3] is methods on a reify, which calls the same method again, or in the most recent case, just return nil. [1]http://www.minecraft.net/docs/NBT.txt [2]https://gist.github.com/761997 [3]http://pastebin.com/AqrsbjuS On Nov 28 2010, 8:14 pm, Zach Tellman ztell...@gmail.com wrote: You're right, that's an omission from the frame syntax. I'll add the ability for all or part of the frame to be scoped as (little- endian ...) and (big-endian ...), with big-endian as the default. Just as a side-note, though, Calx [1] is already handling little- endian data by using encode-to-buffer, where it's writing to a buffer whose endianness has been preset. This obviously isn't a general solution, but just thought I'd point it out. Zach [1]https://github.com/ztellman/calx On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote: If Gloss is to decode incoming packet (byte array) in little-endian format it is straightforward: Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and pass b to decode function that will return Clojure map of decoded values. However, when outgoing packet byte array is to be produced from map of values, encode function will always return ByteBuffer in default big- endian format, so resulting byte array extracted form ByteBuffer using get() method will be incorrect. If Gloss is to support little-endian frames, it seems that endianness needs to be part of frame definition. In that case Gloss decode fun would refuse to accept ByteBuffers with wrong order() and encode fun will always generate the correct result. Zoka On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote: ByteBuffers have an order() method which allows you to toggle the endianness. I haven't tested this, but since everything is built on top of Java's ByteBuffer functionality it should be fine as long as the ByteBuffers are correctly set and correctly ordered with respect to each other. Zach On Nov 23, 2:52 pm, zoka ztomi...@gmail.com wrote: JVM stores numbers in in big endian format - is there a way to process binary stream containing little endian numbers? Zoka On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote: Good question. The solution didn't make the cut for my initial release, but will be added soon. My plan is to have an (ordered- map ...) frame which encodes and decodes the keys in the given order. So for C interop, the frame would be (ordered-map :a :int16, :b :float32) An alternative would be to just turn any vector which is alternating keys and types into an ordered-map, but that seems a bit too magical. Zach On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote: On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote: When writing Calx [1], I discovered it was a huge pain to deal with mixed C datatypes in Java. When writing Aleph [2], I discovered the problem increases by a factor of ten when dealing with streams of bytes. In an attempt to alleviate my own pain, and hopefully help a few other people out, I've written Gloss, which can transform a simple byte-format specification into an encoder and streaming decoder. A full writeup can be found athttps://github.com/ztellman/gloss/wiki. A few people have already asked me how this differs from protocol buffers, so I'll preemptively answer that protocol buffers are a fixed format that cannot be used to interface with external systems. Gloss is
Re: ANN: Gloss, a byte-format DSL
If Gloss is to decode incoming packet (byte array) in little-endian
format it is straightforward:
Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and
pass b to decode function that will return Clojure map of decoded
values.
However, when outgoing packet byte array is to be produced from map of
values, encode function will always return ByteBuffer in default big-
endian format, so resulting byte array extracted form ByteBuffer using
get() method will be incorrect.
If Gloss is to support little-endian frames, it seems that endianness
needs to be part of frame definition. In that case Gloss decode fun
would refuse to accept ByteBuffers with wrong order() and encode fun
will always generate the correct result.
Zoka
On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote:
ByteBuffers have an order() method which allows you to toggle the
endianness. I haven't tested this, but since everything is built on
top of Java's ByteBuffer functionality it should be fine as long as
the ByteBuffers are correctly set and correctly ordered with respect
to each other.
Zach
On Nov 23, 2:52 pm, zoka ztomi...@gmail.com wrote:
JVM stores numbers in in big endian format - is there a way to process
binary stream containing little endian numbers?
Zoka
On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
You're right, that's an omission from the frame syntax. I'll add the
ability for all or part of the frame to be scoped as (little-
endian ...) and (big-endian ...), with big-endian as the default.
Just as a side-note, though, Calx [1] is already handling little-
endian data by using encode-to-buffer, where it's writing to a buffer
whose endianness has been preset. This obviously isn't a general
solution, but just thought I'd point it out.
Zach
[1] https://github.com/ztellman/calx
On Nov 28, 8:50 am, zoka ztomi...@gmail.com wrote:
If Gloss is to decode incoming packet (byte array) in little-endian
format it is straightforward:
Wrap byte array into ByteBuffer b, invoke b.order(LITTLE_ENDIAN) and
pass b to decode function that will return Clojure map of decoded
values.
However, when outgoing packet byte array is to be produced from map of
values, encode function will always return ByteBuffer in default big-
endian format, so resulting byte array extracted form ByteBuffer using
get() method will be incorrect.
If Gloss is to support little-endian frames, it seems that endianness
needs to be part of frame definition. In that case Gloss decode fun
would refuse to accept ByteBuffers with wrong order() and encode fun
will always generate the correct result.
Zoka
On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote:
ByteBuffers have an order() method which allows you to toggle the
endianness. I haven't tested this, but since everything is built on
top of Java's ByteBuffer functionality it should be fine as long as
the ByteBuffers are correctly set and correctly ordered with respect
to each other.
Zach
On Nov 23, 2:52 pm, zoka ztomi...@gmail.com wrote:
JVM stores numbers in in big endian format - is there a way to process
binary stream containing little endian numbers?
Zoka
On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a
simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found
athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a
fixed
format that cannot be used to interface with external systems.
Gloss
is less performant than protocol buffers, but is also much less
picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
The most general case would be a codec that can have mixture of big and little endian fields, so legacy comms protocols or file formats can be supported. For example: (defcodec mixed (ordered-map :b :int16, :a :float32-le)) ; total 6 bytes, first 2 bytes short in big endian format (JVM default), ; followed by 4 byte float in little endian (Intel) format. Zoka On Nov 25, 3:00 am, Zach Tellman ztell...@gmail.com wrote: ByteBuffers have an order() method which allows you to toggle the endianness. I haven't tested this, but since everything is built on top of Java's ByteBuffer functionality it should be fine as long as the ByteBuffers are correctly set and correctly ordered with respect to each other. Zach -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Hi
On 24 November 2010 04:43, Zach Tellman ztell...@gmail.com wrote:
There are a couple of different things being discussed here. I don't
think there's any harm in allowing maps as frames, as long as people
understand that they're arbitrarily ordered (alphabetically by key,
but that's an implementation detail) and that keys specified aren't
optional - if they're not all there, the encoder will fail.
The first issue you raised was that we might want to have finer
control over how maps are encoded, so that we can ensure compatibility
with C structs. This is true, but we still want the decoded versions
of these ordered lists to be maps, because {:a 1, :b 2} is much more
convenient than [:a 1 :b 2] or [[:a 1] [:b 2]].
The second issue you're raising is backwards compatibility. It's
I don't think he's talking about backwards compatibility at all.
Assume you have two C structs that you need to interoperate with. The
first has a few fields and the second has many fields. If you specify
the format with a map, then when you serialise your Clojure data, the
smaller one will create the values in the order you specified in the
map. The larger one will be in some arbitrary order which depends on
Clojure's implementation and will likely not match up with the actual
struct definition.
Or to put it another way, say you have many structs. The first
several have few fields, so you happily use maps to specify them.
Everything works fine. Then you get to a larger struct and you find
that your fields land up in the wrong part of the serialised output.
I haven't looked at Gloss yet, so maybe this isn't possible for some reason?
[...] I wrote the
library mostly to help me write Redis and AMQP clients, but I'm hoping
people will find it useful for a variety of purposes.
I have used Python's struct library a few times (and perl's
pack/unpack once or twice), so I'm sure Gloss would be useful for the
same sorts of things.
--
Michael Wood esiot...@gmail.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
JVM stores numbers in in big endian format - is there a way to process
binary stream containing little endian numbers?
Zoka
On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
On Nov 24, 12:42 am, Michael Wood esiot...@gmail.com wrote:
Hi
On 24 November 2010 04:43, Zach Tellman ztell...@gmail.com wrote:
There are a couple of different things being discussed here. I don't
think there's any harm in allowing maps as frames, as long as people
understand that they're arbitrarily ordered (alphabetically by key,
but that's an implementation detail) and that keys specified aren't
optional - if they're not all there, the encoder will fail.
The first issue you raised was that we might want to have finer
control over how maps are encoded, so that we can ensure compatibility
with C structs. This is true, but we still want the decoded versions
of these ordered lists to be maps, because {:a 1, :b 2} is much more
convenient than [:a 1 :b 2] or [[:a 1] [:b 2]].
The second issue you're raising is backwards compatibility. It's
I don't think he's talking about backwards compatibility at all.
Chris raised the issue of breaking a format by adding keys, but that
will happen no matter how the format is ordered. I just wanted to
make sure that was clear.
Assume you have two C structs that you need to interoperate with. The
first has a few fields and the second has many fields. If you specify
the format with a map, then when you serialise your Clojure data, the
smaller one will create the values in the order you specified in the
map. The larger one will be in some arbitrary order which depends on
Clojure's implementation and will likely not match up with the actual
struct definition.
Or to put it another way, say you have many structs. The first
several have few fields, so you happily use maps to specify them.
Everything works fine. Then you get to a larger struct and you find
that your fields land up in the wrong part of the serialised output.
I haven't looked at Gloss yet, so maybe this isn't possible for some reason?
I feel like I must have done a poor job describing the ordered-map
mechanism earlier. This codec is fine:
(defcodec c {:a :int16, :b :float32})
as long as Gloss is on both sides of the encoding and decoding, or if
the default ordering is coincidentally correct. However, if we want
to invert the position of :a and :b, we use ordered-map:
(defcodec c (ordered-map :b :float32, :a :int16))
This codec still consumes and emits standard Clojure maps, it just
makes sure that the values are encoded in a particular order. The
frame is the only thing that has to change to make the encoding
explicitly ordered; everything else can continue to use unordered
hashes without concern.
I think this solves the problem you describe, but maybe I'm
misunderstanding you.
Zach
[...] I wrote the
library mostly to help me write Redis and AMQP clients, but I'm hoping
people will find it useful for a variety of purposes.
I have used Python's struct library a few times (and perl's
pack/unpack once or twice), so I'm sure Gloss would be useful for the
same sorts of things.
--
Michael Wood esiot...@gmail.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
ByteBuffers have an order() method which allows you to toggle the
endianness. I haven't tested this, but since everything is built on
top of Java's ByteBuffer functionality it should be fine as long as
the ByteBuffers are correctly set and correctly ordered with respect
to each other.
Zach
On Nov 23, 2:52 pm, zoka ztomi...@gmail.com wrote:
JVM stores numbers in in big endian format - is there a way to process
binary stream containing little endian numbers?
Zoka
On Nov 24, 7:24 am, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Neat. License same as Clojure's? Stu When writing Calx [1], I discovered it was a huge pain to deal with mixed C datatypes in Java. When writing Aleph [2], I discovered the problem increases by a factor of ten when dealing with streams of bytes. In an attempt to alleviate my own pain, and hopefully help a few other people out, I've written Gloss, which can transform a simple byte-format specification into an encoder and streaming decoder. A full writeup can be found at https://github.com/ztellman/gloss/wiki. A few people have already asked me how this differs from protocol buffers, so I'll preemptively answer that protocol buffers are a fixed format that cannot be used to interface with external systems. Gloss is less performant than protocol buffers, but is also much less picky about formats. If anyone has any questions, I'd be happy to answer them. Zach [1] https://github.com/ztellman/calx [2] https://github.com/ztellman/aleph -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
This looks a *lot* like something I did a long time ago (Data File Disassembler) when I spent a lot of time reverse-engineering things. I had a view/edit layer on top (On Mac, back when a Mac II was extreme hardware :) and man, was it useful... had pluggable modules for microcontroller disassembly, a few common file formats I needed, etc. Just need to get my Lisp fu back, and I can go back to being productive again. Dave On Tue, Nov 23, 2010 at 12:03 PM, Zach Tellman ztell...@gmail.com wrote: When writing Calx [1], I discovered it was a huge pain to deal with mixed C datatypes in Java. When writing Aleph [2], I discovered the problem increases by a factor of ten when dealing with streams of bytes. In an attempt to alleviate my own pain, and hopefully help a few other people out, I've written Gloss, which can transform a simple byte-format specification into an encoder and streaming decoder. A full writeup can be found at https://github.com/ztellman/gloss/wiki. A few people have already asked me how this differs from protocol buffers, so I'll preemptively answer that protocol buffers are a fixed format that cannot be used to interface with external systems. Gloss is less performant than protocol buffers, but is also much less picky about formats. If anyone has any questions, I'd be happy to answer them. Zach [1] https://github.com/ztellman/calx [2] https://github.com/ztellman/aleph -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Yes. There are Eclipse license headers on all the files, I'll just update the README to reflect that. On Nov 23, 9:51 am, Stuart Halloway stuart.hallo...@gmail.com wrote: Neat. License same as Clojure's? Stu When writing Calx [1], I discovered it was a huge pain to deal with mixed C datatypes in Java. When writing Aleph [2], I discovered the problem increases by a factor of ten when dealing with streams of bytes. In an attempt to alleviate my own pain, and hopefully help a few other people out, I've written Gloss, which can transform a simple byte-format specification into an encoder and streaming decoder. A full writeup can be found athttps://github.com/ztellman/gloss/wiki. A few people have already asked me how this differs from protocol buffers, so I'll preemptively answer that protocol buffers are a fixed format that cannot be used to interface with external systems. Gloss is less performant than protocol buffers, but is also much less picky about formats. If anyone has any questions, I'd be happy to answer them. Zach [1]https://github.com/ztellman/calx [2]https://github.com/ztellman/aleph -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
On Tue, Nov 23, 2010 at 10:03 AM, Zach Tellman ztell...@gmail.com wrote: When writing Calx [1], I discovered it was a huge pain to deal with mixed C datatypes in Java. When writing Aleph [2], I discovered the problem increases by a factor of ten when dealing with streams of bytes. You're not the only one. Thanks for writing this! -- Chris Riddoch -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
I would go for the vector. It's easier than writing (ordered-map ...) all
the time. If you want to generate structs dynamically, you would even simply
create a vector of alternating pairs and do (apply ordered-map my-vec).
Btw: Great library! I think I will rewrite my binary-parser from a project
of mine to use this library!
On Tue, Nov 23, 2010 at 9:24 PM, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
Moritz Ulrich
Programmer, Student, Almost normal Guy
http://www.google.com/profiles/ulrich.moritz
BB5F086F-C798-41D5-B742-494C1E9677E8
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
Looks really good. Thanks for another great library!
On 23 November 2010 21:12, Moritz Ulrich ulrich.mor...@googlemail.comwrote:
I would go for the vector. It's easier than writing (ordered-map ...) all
the time. If you want to generate structs dynamically, you would even simply
create a vector of alternating pairs and do (apply ordered-map my-vec).
Btw: Great library! I think I will rewrite my binary-parser from a project
of mine to use this library!
On Tue, Nov 23, 2010 at 9:24 PM, Zach Tellman ztell...@gmail.com wrote:
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
On Nov 23, 12:03 pm, Zach Tellman ztell...@gmail.com wrote:
When writing Calx [1], I discovered it was a huge pain to deal with
mixed C datatypes in Java. When writing Aleph [2], I discovered the
problem increases by a factor of ten when dealing with streams of
bytes. In an attempt to alleviate my own pain, and hopefully help a
few other people out, I've written Gloss, which can transform a simple
byte-format specification into an encoder and streaming decoder.
A full writeup can be found athttps://github.com/ztellman/gloss/wiki.
A few people have already asked me how this differs from protocol
buffers, so I'll preemptively answer that protocol buffers are a fixed
format that cannot be used to interface with external systems. Gloss
is less performant than protocol buffers, but is also much less picky
about formats.
If anyone has any questions, I'd be happy to answer them.
Looks very useful, Zach. Thanks.
I have a question.
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Thanks,
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
Moritz Ulrich
Programmer, Student, Almost normal Guy
http://www.google.com/profiles/ulrich.moritz
BB5F086F-C798-41D5-B742-494C1E9677E8
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: ANN: Gloss, a byte-format DSL
On Nov 23, 3:24 pm, Zach Tellman ztell...@gmail.com wrote:
On Nov 23, 12:12 pm, Chris Perkins chrisperkin...@gmail.com wrote:
I have only taken a quick look, so maybe I'm misunderstanding the
intent, but it's not clear to me how you would use this for sending
and receiving structured data from, say, a C program.
Taking your example from the wiki:
(def fr (compile-frame {:a :int16, :b :float32}))
Let's say I want to talk to a C program that speaks in structs, like
this:
struct Foo { short a; float b; }
The problem is, the C program cares about order - the short comes
before the float. How does the Clojure program know what order I need
the fields in, since I have specified the format with a map; an
unordered data structure? Is there another way to specify a structure
where order of the fields matters? If so, why have two ways of doing
it? Or am I just missing something?
Good question. The solution didn't make the cut for my initial
release, but will be added soon. My plan is to have an (ordered-
map ...) frame which encodes and decodes the keys in the given order.
So for C interop, the frame would be
(ordered-map :a :int16, :b :float32)
An alternative would be to just turn any vector which is alternating
keys and types into an ordered-map, but that seems a bit too magical.
Zach
I would humbly submit that perhaps you should just go for ordered
formats, always. For example, a vector of two-element vectors could be
the canonical format:
[[:a :int16] [:b :float32]]
Then you could add some macro-sugar to make it pretty:
(struct :a :int16 :b :float32) = (compile-frame [[:a :int16]
[:b :float32]])
The reason I say this is both to simplify (have only one way to
specify a format), because I can envision people getting quite far
down a path of relying on the byte-ordering coming out of compile-
frame, simply due to the fact that clojure uses PersistentArrayMap for
small maps, giving the illusion that they are ordered. Then one day
they confidently add that Nth field to their struct definition (where
N appears to currently be 10), and suddenly they have byte-soup coming
out.
With always-ordered formats, there just seems to be less room for
error.
Just my 2c.
- Chris
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
