Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Friday 06 Mar 2009, David Adams wrote: that claims String.split is about 10x faster than the XML parser - That's the same in almost all languages - it's much faster to use string methods to snip out the bit you want than incur the parsing overhead and XPath look up. However, if you pulling out lots of stuff from the XML, using it as a dataProvider etc., you probably do want it as 'real' XML. -- Tom Chiverton Helping to appropriately promote global second-generation cutting-edge value-added models as part of the IT team of the year, '09 and '08 This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at Halliwells LLP, 3 Hardman Square, Spinningfields, Manchester, M3 3EB. A list of members is available for inspection at the registered office together with a list of those non members who are referred to as partners. We use the word ?partner? to refer to a member of the LLP, or an employee or consultant with equivalent standing and qualifications. Regulated by the Solicitors Regulation Authority. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 2500. For more information about Halliwells LLP visit www.halliwells.com.
[flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
You still avoid telling us your back-end setup. I give up. --- In flexcoders@yahoogroups.com, David Adams dpad...@... wrote: On Fri, Mar 6, 2009 at 5:31 PM, Guy Morton g...@... wrote: You've still managed to avoid telling us about your back-end setup. Tell us more about that and we could probably be of more assistance. No good me suggesting stuff that works for apache on linux to find that you're running openVMS... Sorry, I'm not meaning to be difficult. The server definitely does not support AMF and will not any sooner than...not this year. That's why I've been exploring alternative. Plus, I've spent some insane amount of time over the last ten years studying messages on the wire and can get a bit obsessive (read 'stupid') about sorting out message sizes and round-trip times. I'm hoping to build up a pretty good list of techniques/technologies to compare and put into a test harness. I figure I'll end up using Charles to emulate different relevant bandwidth constraints and it's easy enough to try various payload sizes + encoding time + decoding times.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Sat, Mar 7, 2009 at 2:06 AM, valdhor valdhorli...@embarqmail.com wrote: You still avoid telling us your back-end setup. I give up. I'm not meaning to frustrate you - particularly since you're being kind enough to spend your time sending me suggestions. The back-end is called 4D and it definitely does not have server-side AMF support. I know the right people to ask and have already triple-checked that AMF support is not available. I've even checked with people that might have had reason to roll their own AMF support and none of them have. The main product itself isn't going to have new features added until the next version which doesn't have any announced date. I just didn't think it was worth wasting people's time trying to find an answer on the server-side as I've already got that covered. Thanks for your help.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Sat, Mar 7, 2009 at 8:38 AM, David Adams dpad...@gmail.com wrote: On Sat, Mar 7, 2009 at 2:06 AM, valdhor valdhorli...@embarqmail.com wrote: You still avoid telling us your back-end setup. I give up. I figure I should address the next obvious suggestion in advance. Is something like the following possible? [Flash Player] --- AMF--- [Apache + Zend] --- XML or something--- [4D] Yes, it's entirely possible to put Apache (or what have you) into what amounts to a smart proxying roll. It's even possible this setup would deliver performance gains and other useful features as a bonus. There are definitely times this would be well worth considering, but right now I'm focusing on what can be done without the extra layer of software. I'm well-enough versed in the specifics of this back-end that what I've been trying to sort out is what is native and reasonable in the Flash player. It's on the client side that I'm ignorant. As in so many other areas of my life;-) But, pertinent to this list, I'm ignorant about many aspects of the Flash player and the Flex framework - the framework is huge and many features are easy to overlook. So far it looks like (ignoring back-end): * AMF - Native * XML - Native * JSON - Often used library * Plain-text format - May be able to leverage native features such as String.split() or a custom parser that reads lengths precisely. * Any of the textual formats (XML, JSON, plain-text of whatever sort) may benefit from compression, depending on payload size, compression/decompression time, and network conditions (latency and bandwidth). That's what I've got so far.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
I think (some) people were asking about the back-end to try to help you configure http compression, which is done differently with different servers. Given your requirements, that and XML is probably the way to go for the short term, although simple hand-rolled textual formats may offer much better performance with negligible complexity for some stuff. -Maciek -Original Message- From: David Adams dpad...@gmail.com Reply-To: flexcoders@yahoogroups.com To: flexcoders@yahoogroups.com Subject: Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex? Date: Sat, 7 Mar 2009 08:38:33 +1100 On Sat, Mar 7, 2009 at 2:06 AM, valdhor valdhorli...@embarqmail.com wrote: You still avoid telling us your back-end setup. I give up. I'm not meaning to frustrate you - particularly since you're being kind enough to spend your time sending me suggestions. The back-end is called 4D and it definitely does not have server-side AMF support. I know the right people to ask and have already triple-checked that AMF support is not available. I've even checked with people that might have had reason to roll their own AMF support and none of them have. The main product itself isn't going to have new features added until the next version which doesn't have any announced date. I just didn't think it was worth wasting people's time trying to find an answer on the server-side as I've already got that covered. Thanks for your help.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Sat, Mar 7, 2009 at 9:24 AM, Maciek Sakrejda msakre...@truviso.com wrote: I think (some) people were asking about the back-end to try to help you configure http compression, which is done differently with different servers. Oh. Well, how nice for everyone that I went on at length about completely unrelated details ;-) On the server side, I've had to bolt on a compressor so all I'd really like to know I suppose is what MIME type such files arrive with from another server when everything is working. I've looked up the MIME codes and am not trying to be lazy - I ended up with my files automatically downloading, so I figured I'd ask other people what is actually working for them. Thanks!
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
You don't change the content-type to reflect gzip compression being on on the server. Gzip is a content-encoding and the browser should seamlessly decode it. You do need to set the HTTP headers to tell the browser is gzipped though, eg here's a gzip-encoded response example: Request Accept */* Referer http://www.theage.com.au/ User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1 Response Accept-Ranges bytes Cache-Control max-age=3600, public, proxy-revalidate Connection Keep-Alive Content-Encodinggzip Content-Length 1737 Content-Typeapplication/x-javascript DateFri, 06 Mar 2009 22:37:15 GMT Etag17befe1-111d-775906c0 Expires Fri, 06 Mar 2009 23:37:15 GMT Keep-Alive timeout=2, max=100 Last-Modified Thu, 05 Mar 2009 07:53:39 GMT P3p policyref=http://f2.com.au/w3c/p3p.xml;, CP=CAO DSP LAW CURa ADMa DEVa TAIa PSAa PSDa IVAi IVDi OUR IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT PRE GOV Server Apache VaryAccept-Encoding,User-Agent On 07/03/2009, at 9:30 AM, David Adams wrote: On Sat, Mar 7, 2009 at 9:24 AM, Maciek Sakrejda msakre...@truviso.com wrote: I think (some) people were asking about the back-end to try to help you configure http compression, which is done differently with different servers. Oh. Well, how nice for everyone that I went on at length about completely unrelated details ;-) On the server side, I've had to bolt on a compressor so all I'd really like to know I suppose is what MIME type such files arrive with from another server when everything is working. I've looked up the MIME codes and am not trying to be lazy - I ended up with my files automatically downloading, so I figured I'd ask other people what is actually working for them. Thanks!
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Sat, Mar 7, 2009 at 9:39 AM, Guy Morton g...@alchemy.com.au wrote: You don't change the content-type to reflect gzip compression being on on the server. Gzip is a content-encoding and the browser should seamlessly decode it. You do need to set the HTTP headers to tell the browser is gzipped though, eg here's a gzip-encoded response example: Beautiful. Thanks very much for the headers.
[flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
What kind of server do you have? I was fairly sure there was an implementation of AMF for every server out there. --- In flexcoders@yahoogroups.com, David Adams dpad...@... wrote: * Use AMF. Can't. It's not supported on the server and a project constraint is thou shalt not implement binary protocols by hand.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Fri, Mar 6, 2009 at 12:48 AM, valdhor valdhorli...@embarqmail.com wrote: What kind of server do you have? I was fairly sure there was an implementation of AMF for every server out there. Nope. I can bolt in zip/unzip and a JSON encoder/decoder - but not AMF. Speaing of which, I think I've asked my question badly enough that no one thought to suggest JSON. I guess my most general question is: What techniques can I explore for reducing message sizes that work efficiently in Flex? As I understand it: -- AMF is native fast. -- JSON is not native I'm not sure how fast it is. -- XML is native and relatively fast...for XML. -- Unzipping is native and fast...but not many methods are supported. -- Custom binary protocols are fine as long as I draw the data down into a general object or ByteArray and then parse it myself. Of course, that means the parser is running in ActionScript - not code compiled into the player. Any other ideas? Thanks, as always, for any suggestions.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Mon, Mar 2, 2009 at 2:34 AM, Amy amyblankens...@bellsouth.net wrote: Plain text. Further to this suggestion, I just stumbled across a note from 2007 that claims String.split is about 10x faster than the XML parser - making delimited ASCII a good solution for (as an example) transferring name-value pairs: http://onflash.org/ted/2007/11/apis-with-delimited-ascii-text-vs-xml.php As noted, gzip compression can reduce the overall message size. I'm looking forward to the moment when I finally test and compare various solutions - I'll post results whenever I've got them.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
I find it hard to believe you are running a back end that's incapable of sending AMF. I thought perl was problematic for AMF...but even it has a solution that works...so what *are* you using on the backend? You running a VAX machine or something? :-) JSON isn't binary but is more compact than XML. OTOH, parsing it may not be as quick as XML. I wouldn't be manually zipping and unzipping data - a better idea would be to enable GZIP on the web server and let the browser do the work (not that that is trouble-free but still...). Here: http://www.jamesward.com/blog/2007/12/12/blazebench-why-you-want-amf-and-blazeds/ That should help you see what combinations are fast. Guy On 06/03/2009, at 8:31 AM, David Adams wrote: On Fri, Mar 6, 2009 at 12:48 AM, valdhor valdhorli...@embarqmail.com wrote: What kind of server do you have? I was fairly sure there was an implementation of AMF for every server out there. Nope. I can bolt in zip/unzip and a JSON encoder/decoder - but not AMF. Speaing of which, I think I've asked my question badly enough that no one thought to suggest JSON. I guess my most general question is: What techniques can I explore for reducing message sizes that work efficiently in Flex? As I understand it: -- AMF is native fast. -- JSON is not native I'm not sure how fast it is. -- XML is native and relatively fast...for XML. -- Unzipping is native and fast...but not many methods are supported. -- Custom binary protocols are fine as long as I draw the data down into a general object or ByteArray and then parse it myself. Of course, that means the parser is running in ActionScript - not code compiled into the player. Any other ideas? Thanks, as always, for any suggestions.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Fri, Mar 6, 2009 at 11:55 AM, Guy Morton g...@alchemy.com.au wrote: JSON isn't binary but is more compact than XML. OTOH, parsing it may not be as quick as XML. I haven't tested it yet in the Flash player but the post I linked to earlier claimed String.split (native to the player) is considerably faster than the native XML parser - at least in some cases. That's not hard to imagine, really. Either JSON or XML are text - both of which should benefit - even quite dramatically - from compression. I wouldn't be manually zipping and unzipping data - a better idea would be to enable GZIP on the web server and let the browser do the work (not that that is trouble-free but still...). That sounds great and I'd like to pursue this as a few other people have made the same observation. In my tests, the gzipping wasn't happing in the browser prior to the data being loaded into the Flash player. I set the MIME type on the server to application/gzip or similar (I looked up the correct-sounding type at the time.) Can anyone point me to a working example of gzipping working 'passively' but effectively? Namely, that the Flash player gets uncompressed data even though the server sends the data compressed. That would be ideal. Here: http://www.jamesward.com/blog/2007/12/12/blazebench-why-you-want-amf-and-blazeds/ That should help you see what combinations are fast. Thanks - it's a great article. The author mentions gzipping but I haven't found the specifics. I suspect he's comparing processes without any compression.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
You've still managed to avoid telling us about your back-end setup. Tell us more about that and we could probably be of more assistance. No good me suggesting stuff that works for apache on linux to find that you're running openVMS... On 06/03/2009, at 4:12 PM, David Adams wrote: On Fri, Mar 6, 2009 at 11:55 AM, Guy Morton g...@alchemy.com.au wrote: JSON isn't binary but is more compact than XML. OTOH, parsing it may not be as quick as XML. I haven't tested it yet in the Flash player but the post I linked to earlier claimed String.split (native to the player) is considerably faster than the native XML parser - at least in some cases. That's not hard to imagine, really. Either JSON or XML are text - both of which should benefit - even quite dramatically - from compression. I wouldn't be manually zipping and unzipping data - a better idea would be to enable GZIP on the web server and let the browser do the work (not that that is trouble-free but still...). That sounds great and I'd like to pursue this as a few other people have made the same observation. In my tests, the gzipping wasn't happing in the browser prior to the data being loaded into the Flash player. I set the MIME type on the server to application/gzip or similar (I looked up the correct-sounding type at the time.) Can anyone point me to a working example of gzipping working 'passively' but effectively? Namely, that the Flash player gets uncompressed data even though the server sends the data compressed. That would be ideal. Here: http://www.jamesward.com/blog/2007/12/12/blazebench-why-you-want-amf-and-blazeds/ That should help you see what combinations are fast. Thanks - it's a great article. The author mentions gzipping but I haven't found the specifics. I suspect he's comparing processes without any compression.
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Fri, Mar 6, 2009 at 5:31 PM, Guy Morton g...@alchemy.com.au wrote: You've still managed to avoid telling us about your back-end setup. Tell us more about that and we could probably be of more assistance. No good me suggesting stuff that works for apache on linux to find that you're running openVMS... Sorry, I'm not meaning to be difficult. The server definitely does not support AMF and will not any sooner than...not this year. That's why I've been exploring alternative. Plus, I've spent some insane amount of time over the last ten years studying messages on the wire and can get a bit obsessive (read 'stupid') about sorting out message sizes and round-trip times. I'm hoping to build up a pretty good list of techniques/technologies to compare and put into a test harness. I figure I'll end up using Charles to emulate different relevant bandwidth constraints and it's easy enough to try various payload sizes + encoding time + decoding times.
[flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
--- In flexcoders@yahoogroups.com, David Adams dpad...@... wrote: I'm working on a system that transfers data from a back-end to Flex for display and manipulation. In this case, I can control the message format on the server-side. I'm pretty happy sending XML because of how easy it is to work with XML in Flex. On the other hand, XML is sometimes ridiculously inefficient as a message format as it tends to be pretty bloated. I'm imagining a few strategies: * Strip and reduce the XML to use stupidly short element names and no spare white-space to reduce the message size a bit. This is a weak option in a lot of ways but it's easy to do. * Compress the XML and decompress it in Flex. This is an appealing option as I get to work with the XML but don't have such big messages. Unfortunately, I haven't been able to get this to work using zlib and ByteArray.uncompress(). I'll also assume that there is some kind of sweet-spot when the cost of compression/decompression outweighs the cost of the larger message. Once I get this working, I could try to figure out when compression is worth engaging and then have the server compress when it (guesses) it will make sense. * Use AMF. Can't. It's not supported on the server and a project constraint is thou shalt not implement binary protocols by hand. * Use a custom binary format, package the data that way on the server, and then write a custom parser in Flex. Can't. See previous point and another project constraint, thou shalt not invent new binary formats. * Use another binary format that Flex does understand. Are there any? Basically, I'm looking for an efficient message transfer format that I don't have to invent or hand-code. Plain text. Long before the invention of XML, developers used csv or csv-like files to power their data-driven apps. You'll have to load the plain text into objects, but hopefully you don't have a constraint against that :o)
Re: [flexcoders] Re: Compressing messages/data: What binary/compression formats are workable in Flex?
On Mon, Mar 2, 2009 at 2:34 AM, Amy amyblankens...@bellsouth.net wrote: Plain text. Long before the invention of XML, developers used csv or csv-like files to power their data-driven apps. You'll have to load the plain text into objects, but hopefully you don't have a constraint against that :o) Thanks for the suggestion. Sure, I can do some kind of a plain-text format -- Fixed-length (not likely) -- Variable length with length bytes -- Some kind of delimited system (csv or what have you) Depending on the content, -- Fixed length is a bad idea...but it's useful in other contexts. (For example, if you have a 'natural index' that lets you calculate a rows position - then you can grab a row out of a huge file with a couple of reads... but that's not my problem.) -- Variable-length with a length byte: This would add only one byte per variable-length item, so it's about as efficient as I can get. -- Delimited system: Not as efficient as the previous method, but popular so easy to support in many ways. What I'm compariing against are two things: 1) The naive 'stupid unreadable XML' format that lets me keep using XML but reduces the message size a bit. 2) True compression/binary format. The unreadable XML approach reduces message size by making element names shorter and removing whitespace. For example, instead of ResultSet Record FirstNameBob/FirstName /Record /ResultSet ...you get something like this: abcBob/c/b/a This obviously throws away the advantages of XML as 'human readable markup'...an advantage that is irrelevant in my case - I'm just trying to send messages that are produced by one program and read by another program - no humans involved. What I'm really hoping for is real compression - as plain text or XML tend to compress fairly dramatically. Either that, or some other binary format this is not AMF and that is supported natively in Flash. Sorry for the length of my answer...I just suck at writing short messages when I'm sorting through a problem. And, again, thanks for the help.
