The danger of just testing the JS code is that the JS code is only as good as we think it is
Yeah, absolutely agree, it would definitely be best to test/validate against the 'reference' implementation :) We do need to address this in general, and I am prepared to put the time in, but for now I need to get something working asap for Carlos. So I will stay in AMFBinaryData with something that already works on the deserialization side and is semi-complex (2-3 levels deep, includes internal referencing, an AC, etc). If I can find and fix the problem on the serialization side for something that deserializes well in AMFBinaryData, with the example that I have, then I think it is progress. Thanks for your input Alex, much appreciated. -Greg On Wed, Nov 21, 2018 at 9:00 AM Alex Harui <aha...@adobe.com.invalid> wrote: > Ah, I see. It looks like the property list is a sequence of strings, so > it should be possible to determine what Flash thinks it is, by examining > the byte array dump and then use exactly that order in AMFBinaryData. The > danger of just testing the JS code is that the JS code is only as good as > we think it is, whereas what Flash is doing is the "standard". SWFDump may > also reveal the traits in the correct order, not sure. > > We could try to understand how Flash orders traits and then write the > traits in order to REFLECTION_INFO, but I don't think that would be worth > the effort. > > Good luck, > -Alex > > On 11/20/18, 11:49 AM, "Greg Dove" <greg.d...@gmail.com> wrote: > > On Wed, Nov 21, 2018 at 8:43 AM Alex Harui <aha...@adobe.com.invalid> > wrote: > > > Regarding sorting properties: The spec implies that sorting > properties > > is not required. The only thing that matters is that the bytes for > the > > values are in the same order as the traits property list. > > > > Yes, I understand that. I only need to do it for testing: so that I > can > compare apples with apples in terms of the how the object referencing > 'tables' work when encoding and decoding. > I can do this with round-tripping in AMFBinaryData, but I have no > control > over that with ByteArray, for example, so it makes it very difficult to > compare the outputs between the two implementations for complex stuff. > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.adobe.com%2Fcontent%2Fdam%2Facom%2Fen%2Fdevnet%2Fpdf%2Famf-file-format-spec.pdf&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636783401968289581&sdata=19S0xtlwGbg5ZhutUxT%2FhkTI1WcDcU3AbttMDRXFQ8E%3D&reserved=0 > > > > HTH, > > -Alex > > > > On 11/20/18, 11:31 AM, "Greg Dove" <greg.d...@gmail.com> wrote: > > > > On Wed, Nov 21, 2018 at 7:52 AM Alex Harui > <aha...@adobe.com.invalid> > > wrote: > > > > > In snooping around the code a bit, I noticed: > > > > > > 1) Flex MX ArrayCollections usually wraps an ArrayList and thus > > actually > > > justs reads/writes an Array. So, in theory, the bytes that > you find > > when > > > serializing an Array vs ArrayCollection should only differ by > the > > alias > > > stored (and maybe traits). > > > > > > Yes, that's what I meant when I said that in terms of > > 'externalizable' it > > seems quite 'easy'. I'm actually mapping the alias to royale > ArrayList > > and > > have this working in a special case setup for now on the inbound > side. > > My > > intention is to make something general that is 'Externalizable' > for > > royale > > though. > > > > > > > 2) AMFBinaryData has some special casing for ArrayCollection > > already, and > > > probably shouldn't. IMO, that special handling was probably > to allow > > > reading of an AC and ending up with an Array. We don't want > that in > > Royale. > > > > > > It doesn't really do anything as it was. It just arrives as > Array > > (which > > was usable, but not ideal) - this side seems easier to fix. But > it > > doesn't > > handle sending yet, that's the main focus for me right now. > > > > > > > 3) AMFBInaryData implies that there is an externalizable bit > in the > > byte > > > stream. However, it is not set to true in writeTraits and > probably > > should > > > be, and further, the property list should probably not be > written to > > the > > > stream if the class is IExternalizable. The API for > IExternalizable > > > implies that the implementation "knows" the order of > properties. > > > > > > Yes I have no properties for the AC output - if > Externalizable, the > > properties are ignored > > but I will double check I am setting the externalizable marker > bit > > correctly- thanks > > > > > > > > > One other thought (you've may have already realized this): > You may > > not > > > need RemoteObject and a server to test IExternalizable. You > could > > write > > > a Flash SWF that writes to ByteArray and then dumps the actual > bytes > > and > > > compare to a JS version using AMFBinaryData. I think the bytes > > should be > > > exactly the same. > > > > > > > Yes, I did something like this already, although it was with a > > variation of > > AMFBinaryData that works in swf instead of javascript, and not > with > > exactly > > the same complex content yet, so far just testing combinations of > > ArrayCollections, including reference copies.... I may go back to > > that, but > > I thought I would start again by round-tripping something that > decodes > > well > > already in the javascript version, because I am planning to use > sort() > > on > > the properties for reading and writing , which should > theoretically > > make > > the virtual object tables the same for reader and writer - this > is the > > part > > that becomes hard to compare between the implementations with > complex > > graphs, because the indexing order for object references will > change > > depending on the order that they are encountered when writing > > (properties > > order) as they are recursed - and so makes 'actual bytes' > comparisons > > not > > easy between different implementations. (Hopefully that makes > sense) > > > > btw The avmplus code has been at least helpful already to catch > another > > thing that is missing in the code we have :) > > if a function is assigned as a value to a property (variable or > > accessor > > with Function type ) then that value needs to be written as > undefined > > > > > > > > > > > HTH, > > > -Alex > > > > > > On 11/20/18, 10:14 AM, "Greg Dove" <greg.d...@gmail.com> > wrote: > > > > > > Thanks guys, > > > > > > I appreciate any guidance. I will take a look at the > avmplus > > code. > > > BlazeDS > > > has a bunch of code that special-cases collection handling > so I > > was > > > looking > > > for something that might be closer to how the player does > it. > > But if I > > > need > > > to I will go back there. > > > > > > All AC does is say 'I'm external' and then outputs the > internal > > source > > > array. So it seems 'easy' but I think the object tracking > is > > providing > > > wrong indexing for subsequent readers. > > > > > > So far (and I guess the caveat there is that I have not > been > > receiving > > > really complex objects) the reading side seems fine, it is > the > > writing > > > where I getting bad results. I need to make sure the > 'virtual' > > > reference > > > tables that the future reader uses have correct index > references. > > > Anyway, I do have some semi-complex content (a user object > with > > roles > > > and > > > other related subobjects) deserializing correctly with AC > being > > > referenced > > > correctly in the final deserialized object, and it has some > > internal > > > object > > > references, so I will start by trying to round-trip that > through > > > serialization/deserialization in local AMFBinaryData and > look > > for more > > > clues. But I will alos check the avmplus code first - > thanks! > > > > > > Cheers, > > > Greg > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:56 AM Alex Harui > > <aha...@adobe.com.invalid> > > > wrote: > > > > > > > This is a major US holiday week, so not a good week for > trying > > to get > > > > internal documents released. So looking at the code in > > AVMPlus is > > > probably > > > > as good as it will get. > > > > > > > > I could be wrong, but I would think IExternalizable > would not > > affect > > > the > > > > byte stream very much at all. I would think its only > impact > > would > > > be to > > > > pass the bytes to the implementing class. The goal, > AIUI, > > isn't to > > > encode > > > > an ArrayCollection into the bytes, it is to take a > subset of > > the AC > > > > properties and put them in the stream so that the > > deserialization > > > code in > > > > BlazeDS and AMFPHP can decode it into whatever class > > represents AC > > > on the > > > > server. So not only do you have AVMPlus code to > reference, I > > think > > > there > > > > may be code in BlazeDS. > > > > > > > > I would think the basic AMF deserialization logic > currently is: > > > > > > > > ReadAlias (of the first class in the stream) > > > > GetClassFromAlias > > > > New Class > > > > Read property list of class. > > > > readAlias (the type of the first property in the > property list > > > > GetClassFromAlias > > > > New Class (to get an object that is the value of that > first > > property) > > > > Recursively deserialize that object > > > > Set the first property to that object > > > > Loop. > > > > > > > > IMO, IExternalizable is only supposed to break out of > this > > pattern > > > and > > > > call the implementation to recursively deserialize the > object. > > > > > > > > I think the code just needs to add a check after > > getClassFromAlias > > > that > > > > determines if the instance is IExternalizable and then > skips > > getting > > > the > > > > property list and just calls readExternal. > > > > > > > > Of course, I could be wrong... > > > > -Alex > > > > > > > > > > > > > > > > On 11/20/18, 3:48 AM, "Frost, Andrew" > <andrew.fr...@harman.com > > > > > > wrote: > > > > > > > > Hi guys > > > > > > > > I'm not 100% sure whether this will help, but if I'm > right > > in > > > thinking > > > > that what you're trying to do is to mimic the Flash > object > > > serialisation, > > > > then I think that this information is actually in the > public > > domain > > > > already.. > > > > > > > > Carlos, you wrote: > > > > "ArrayCollection uses IExternalizable and that part > is our > > main > > > > problem since we don’t have references, just simple docs > but > > no real > > > > implementation, guides or notes about it." > > > > > > > > The IExternalizable.as implementation is part of > avmplus: > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fadobe%2Favmplus%2Fblob%2Fmaster%2Fcore%2FIExternalizable.as&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636783401968289581&sdata=um%2B%2FLxWKYkQZPWdnLYlnKyHMns%2FeA5estHXMp53t6X0%3D&reserved=0 > > > > ArrayCollection is of course part of Flex: > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflex-sdk%2Fblob%2Fmaster%2Fframeworks%2Fprojects%2Fframework%2Fsrc%2Fmx%2Fcollections%2FArrayCollection.as&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636783401968289581&sdata=aJCV%2BCxznStnOpknR3%2FdJyBjZD%2FtXbZ5mEkLsJndRS0%3D&reserved=0 > > > > and you can then follow through what's happening. > > > > > > > > And, having just done that, I'm guessing you're > looking at > > how > > > > "readObject" works. Presumably you're serialising into a > > ByteArray, > > > which > > > > means ultimately you'll end up in the heart of the > avmplus > > code: > > > > > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fadobe%2Favmplus%2Fblob%2Fmaster%2Fcore%2FToplevel.cpp%23L1504&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636783401968289581&sdata=xRk0Cv3KABZvxrbovqOJ6sXObrlYjCggBDj%2FXRiAeJA%3D&reserved=0 > > > > and > > > > > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fadobe%2Favmplus%2Fblob%2Fmaster%2Fcore%2FAvmSerializer.cpp&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636783401968289581&sdata=oHh1CCs3XEXzfjPCvHU2kWH0zjH5Wi7xA8UCj57wJDs%3D&reserved=0 > > > > which is quite complex...! > > > > > > > > Not sure if those help at all but it looks to me > like it > > should > > > be > > > > possible to have some JavaScript code to do all this. If > this > > is the > > > sort > > > > of area that you're looking at then let me know, I've got > > extensive > > > > experience with the avmplus code and have builds of it > that I > > can > > > > tweak/play with to debug what happens in various > scenarios! > > but I > > > don't > > > > have any experience with using RemoteObject hence this > may all > > be > > > > completely irrelevant.. > > > > > > > > thanks > > > > > > > > Andrew > > > > > > > > > > > > -----Original Message----- > > > > From: Greg Dove [mailto:greg.d...@gmail.com] > > > > Sent: 20 November 2018 09:47 > > > > To: dev@royale.apache.org > > > > Subject: [EXTERNAL] Re: Problems sending MX > > ArrayCollections > > > with MX > > > > RemoteObject > > > > > > > > Hi Alex, > > > > > > > > To follow on from what Carlos explained... your > initial > > > > implementation with AMFBinaryData was working quite well > for > > smaller > > > object > > > > graphs. > > > > > > > > And I was able to make some changes to get it to > > special-case > > > > ArrayCollection for now on the reading side (it's not a > proper > > > > implementation, but it was to get things working for > now). > > > > > > > > What I am missing is some good implementation notes > > ('conforming > > > > readers and writers must do this'). For example, I'm not > sure > > what > > > the > > > > object table > > > > (rememberObject/traitsByReference) rules are for > > > Externalizables, and > > > > it's really hard to 'discover' through comparisons with > things > > like > > > > ByteArray or BlazeDS responses partly because the > undefined > > output > > > order of > > > > the fields in a large graph which makes direct > comparison of > > byte > > > sequences > > > > from the 2 sources not so easy. > > > > > > > > In the end I will be happy to keep working on getting > > > AMFBinaryData > > > > fully featured in my own time to help with the general MX > > effort, but > > > > Carlos needs certain things working for his project, > which are > > our > > > current > > > > priorities. > > > > > > > > I do have some improvements that I can get in for > > AMFBinaryData > > > in the > > > > next few days (I'm still refining things): > > > > I have some code that avoids inappropriate field > data for > > > > serialization: > > > > only accessors with readwrite access (not getter > only or > > setter > > > only) > > > > no static accessors or variables excludes any field with > > 'Transient' > > > > metadata > > > > > > > > > > > > But I'm still actively trying to figure out how to > get the > > AC > > > writing > > > > correctly in a complex graph, and I really need to > understand > > what > > > the > > > > rules are supposed to be. In terms of 'Externalizable' AC > > seems quite > > > > simple, so I thought it would be easy... but I think I am > > messing up > > > the > > > > object reference table when I do this... I was hoping > there > > might be > > > > something documented for implentation notes that is > 'public' > > but > > > 'not well > > > > publicized' that you might be aware of, perhaps through > some > > of your > > > > internal connections. if we can get this fully fledged as > > quickly as > > > > possible, even if it just focused on AC for now on the > > > 'Externalizable' > > > > side then that will help Alina and others as well, I > > think. I've > > > had > > > > my head in this for a bit now, so if I have the info, I > will > > find > > > the time. > > > > > > > > Hopefully you can provide some insight! > > > > -Greg > > > > > > > > > > > > > > > > > > > > On Tue, Nov 20, 2018 at 10:19 PM Carlos Rovira < > > > > carlosrov...@apache.org> > > > > wrote: > > > > > > > > > Hi Alex, > > > > > > > > > > Greg and I are working this days in sending MX > > ArrayCollection > > > with > > > > MX > > > > > RemoteObject to BlazeDS. We found that part is not > > working. > > > > > This is huge problem since without solving this we > can't > > > progress in > > > > > our migration, so hope you could take care of this > issue > > and > > > help us > > > > > to find the solution. > > > > > > > > > > The main problem is that we don't have or find a > good > > > reference of > > > > the > > > > > way it should be implemented and the format makes > it > > difficult > > > to > > > > discover. > > > > > ArrayCollection uses IExternalizable and that part > is > > our main > > > > problem > > > > > since we don’t have references, just simple docs > but no > > real > > > > > implementation, guides or notes about it. > > > > > > > > > > Our better bet is that you could get some help at > Adobe > > > internally > > > > > since I don't think this info is in internet (at > least we > > > search > > > > > heavily to try to find). > > > > > > > > > > I think with out this problem fixed no application > using > > MX > > > > > RemoteObject for communication could be migrated > from > > Flex to > > > > Royale, > > > > > since as soon they want to send and complex graph > that > > uses MX > > > > ArrayCollection it will fail. > > > > > > > > > > Greg can comment more about this problem since he > was > > working > > > hard > > > > on > > > > > the issue this days. > > > > > > > > > > Thanks in advance for your help > > > > > > > > > > > > > > > -- > > > > > Carlos Rovira > > > > > > > > > > > > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclicktime.symantec.com%2Fa%2F1%2FfECozPwEKwcw-Ikf3a3HC4wIeBurGWwaEzA&data=02%7C01%7Caharui%40adobe.com%7C803c84453c6d462e43b308d64f2158a2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636783401968289581&sdata=m5l%2BCrLCKoSQKdA6Yn30C3q9C70tUPs%2Bi85%2BsbuCIkU%3D&reserved=0 > > > > > > > > > > > > > > > E2QRfLRk=?d=ezg34mnblPal5Fyvz8kFGVPZYWSwgwTIshTP4R6ZZBwOU_9DnOO0vulBQk > > > > > > > > > > > > > > > WMjrartx1QiEcmaQF0W-s21xYUmmwd8VZI2paWYczV2yv9_RGOX7HvcYTbT5-boGyLZsXs > > > > > > > > > > > > > > > i81oDbXjP_kE8HOHv-CO0RmzekCJ4IXZ1dfI4iczGVVKJlFs3Z9QfZKPIsqL8bfsOPw83b > > > > > > > > > > > > > > > 72ng2yLuRO4g252wF9TWPtu2CAYr-VXOzqqiXrKi2Zrk9wM3FFEc9hUJ9dpb5b4bflw_r8 > > > > > > > > > > > > > > > nj8z30JVYIfUlmah923Rr-8mDPJO0JvmMDiSEWJojxBQAi_wWktydSWyDvGSDJidxjtFsq > > > > > > > > > > > > > > > UpjRFuCNjuBFxim5GH2bZCj-0Gra3yfagBCVNYiT95Zq5loJN_-kHI8W8-jEOaVK-3Hgd4 > > > > > > > > > > > > > > > iAgxK7oB4xr5-PopRVTJDHsb3OeOqtd9_rjFDbCq0EtWEowCy2xcWbcBzo0IiQhRTv3Mrs > > > > > x48H3qjmo%3D&u=http%3A%2F%2Fabout.me%2Fcarlosrovira > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
