Last Saturday I promised to Chris I will confirm the case and I just did.

As suggested - I rearranged field order in State type and made a test to
proof myself, that it works:
https://github.com/apache/plc4x/commit/3010fdd15524410dfd55746562ed44bcee0fe80f

I've closed related issue (PLC4X-193) and created pull requests. Changes
in mspec are not needed for that case.

It took quite long, but we managed to find solution.

Cheers,
Łukasz


On 01.05.2020 10:53, Christofer Dutz wrote:
> Hi folks,
> 
> so Lukasz just sent me a little pcap file with some AMS packets in there and 
> my assumption was correct:
> 
> for example in the byte-stream I can see the "flags" being 0x0400 ... However 
> wireshark read this as unsigned short LE and therefore re-aranged the bytes:
>     StateFlags: 0x0004
>         .... .... .... ...0 = RESPONSE: Not set
>         .... .... .... ..0. = NO RETURN: Not set
>         .... .... .... .1.. = ADS COMMAND: Set
>         .... .... .... 0... = SYSTEM COMMAND: Not set
>         .... .... ...0 .... = HIGH PRIORITY COMMAND: Not set
>         .... .... ..0. .... = TIMESTAMP ADDED: Not set
>         .... .... .0.. .... = UDP COMMAND: Not set
>         .... .... 0... .... = INIT COMMAND: Not set
>         0... .... .... .... = BROADCAST: Not set
> 
> So in above example only one field is true: "ADS COMMAND" ... so if I decode 
> 0x0400 to binary that’s: 00000100 00000000
> 
> So this exactly fits my proposed mspec definition of: 
> 
>     [type 'State'
>             [simple     bit 'initCommand'           ]
>             [simple     bit 'updCommand'            ]
>             [simple     bit 'timestampAdded'        ]
>             [simple     bit 'highPriorityCommand'   ]
>             [simple     bit 'systemCommand'         ]
>             [simple     bit 'adsCommand'            ]
>             [simple     bit 'noReturn'              ]
>             [simple     bit 'response'              ]
>             [simple     bit 'broadcast'             ]
>             [reserved   int 7 '0x0'                 ]
>         ]
> 
> So I strongly object introducing any special endianness "feature" for this 
> (in this case) ... 
> we might encounter something where we need it, but for this we don't.
> 
> Chris
> 
> 
> 
> Am 30.04.20, 23:02 schrieb "Christofer Dutz" <christofer.d...@c-ware.de>:
> 
>     Hi Lukasz,
> 
>     I wasn't suggesting to look how WireShark decodes things. Cause I would 
> assume that they read a short in little endian and then (on the rearranged 
> bytes) apply some bit checks. 
> 
>     I was more suggesting that you look into the bytes Wireshark shows you 
> and to check their exact position.
> 
>     When implementing drivers, I used the Mac's Calculator tool quite often 
> to see the bits for a given byte value.
> 
>     In your example I can't see any need for endianness at all ... it's all 
> just bits and one empty block which forms a full byte together with the first 
> bit.
>     As this block doesn't cross any byte boundaries I can't see any problems 
> with this.
> 
>     Assuming I understand what you are implying with 
> 
>     [type little endian 'State'
>             [simple     bit 'broadcast'             ]
>             [reserved   int 7 '0x0'                 ]
>             [simple     bit 'initCommand'           ]
>             [simple     bit 'updCommand'            ]
>             [simple     bit 'timestampAdded'        ]
>             [simple     bit 'highPriorityCommand'   ]
>             [simple     bit 'systemCommand'         ]
>             [simple     bit 'adsCommand'            ]
>             [simple     bit 'noReturn'              ]
>             [simple     bit 'response'              ]
>         ]
> 
>     Then it should be equal to:
> 
>     [type 'State'
>             [simple     bit 'initCommand'           ]
>             [simple     bit 'updCommand'            ]
>             [simple     bit 'timestampAdded'        ]
>             [simple     bit 'highPriorityCommand'   ]
>             [simple     bit 'systemCommand'         ]
>             [simple     bit 'adsCommand'            ]
>             [simple     bit 'noReturn'              ]
>             [simple     bit 'response'              ]
>             [simple     bit 'broadcast'             ]
>             [reserved   int 7 '0x0'                 ]
>         ]
> 
>     Which also seems to be the exact order how the bits go over the wire ... 
> I mean ... it doesn't make any sense to put the 7 empty bits in the middle.
> 
>     Chris
> 
> 
>     Am 30.04.20, 22:42 schrieb "Łukasz Dywicki" <l...@code-house.org>:
> 
>         Hey Chris and others,
>         I let this topic to dust and freeze a bit, however I would like to 
> lift
>         it up again. Since the 0.7 release is now in discussion and ADS is not
>         ported yet I believe it is reasonable to validate this problem.
> 
>         As you suggested I've checked wireshark sources to see how they parse
>         state fragment of AMS header. While I can not read C properly and was
>         unable to locate actual reading logic, I do see they care about 
> endianness:
>         
> https://github.com/wireshark/wireshark/blob/5cf3fd03f1538b37704d83b6c38b8223f9036108/plugins/epan/ethercat/packet-ams.c#L429
> 
>         Their frame traversal goes in following order:
>         - response
>         - no return
>         - ads cmd
>         - system cmd
>         - high priority
>         - timestamp add
>         - udp
>         - init cmd
>         - broadcast
> 
>         When I've tried to model mspec with respect to little endian order, I
>         end up with following field order:
>         - init cmd
>         - udp
>         - timestamp add
>         - high priority
>         - system cmd
>         - ads cmd
>         - no return
>         - response
>         - [reserved]
>         - broadcast
> 
>         I haven't seen ADS protocol book, my main source of information is
>         infosys which is rather short in this area. I doubt if they would
>         describe fields in above order.
> 
>         My initial attempts were similar to what you said - to read the value
>         and then shift bytes. With your feedback I started to look for better
>         ways to solve it. With my last attempt I solved the problem by
>         re-adjusting fields during code generation, so they appear in a valid
>         byte order. Beside additional flag at the mspec there is a need for 
> this
>         helper:
>         
> build-utils/protocol-base-mspec/src/main/java/org/apache/plc4x/plugins/codegenerator/language/mspec/model/definitions/DefaultComplexTypeDefinition.java
> 
>         In above commit the State declaration in the mspec looks like this:
>         [type little endian 'State'
>             [simple     bit 'broadcast'             ]
>             [reserved   int 7 '0x0'                 ]
>             [simple     bit 'initCommand'           ]
>             [simple     bit 'updCommand'            ]
>             [simple     bit 'timestampAdded'        ]
>             [simple     bit 'highPriorityCommand'   ]
>             [simple     bit 'systemCommand'         ]
>             [simple     bit 'adsCommand'            ]
>             [simple     bit 'noReturn'              ]
>             [simple     bit 'response'              ]
>         ]
> 
>         Maybe it is not in line with wireshark, but it is consistent and:
>         1) requires minor change in code generator
>         2) does not involve additional processing of data
>         3) mspec is still easy to write
>         4) type declaration is in line with wireshark output when reading from
>         the left and not from the top. ;-)
>         5) it can be adjusted to be 100% consistent with wireshark by changing
>         helper logic. ;-)
> 
>         Best regards,
>         Łukasz
> 
> 
>         On 14.04.2020 21:25, Christofer Dutz wrote:
>         > Hi Lukasz,
>         > 
>         > I see it differently.
>         > 
>         > If you say something is LE/BE then it should be consistent. 
>         > 
>         > I for my part like it if the mspec is consistent with what I see in 
> WireShark. Otherwise you would always have to do a sort of 2 step parsing:
>         > 
>         > "Read 4 bytes in BE and then on the rearranged byte sequence take 
> the first 3 bits and call them X and the next bit after that Y and so on."
>         > 
>         > In this case you would also have to know how large the chunk is to 
> read in order to know how to decode it's content.
>         > Having a historically partitioned set of two 16 bit bit-fields 
> would look completely different to a 32 bit bit-field. 
>         > 
>         > I don't like adding this extra level, even if it's probably the way 
> protocols were implemented when lacking the ability to write individual bits.
>         > I guess the usual process was to read a BE short (2 bytes) and to 
> the resulting unsigned short value apply different logical or operations to 
> get the information.
>         > In PLC4X we can simply read any number of bits and don't have to 
> twist our minds in which chunks I have to read things first. 
>         > 
>         > So I think the way you are proposing is probably the wider used 
> version due to legacy restrictions, but also the more complicated way.
>         > 
>         > So I would like to keep it simple, even if the developer will have 
> to do a little more brain-work. 
>         > 
>         > I mean all we are doing in this project is going the extra mile so 
> others don't have to ... at least the ones contributing are doing that.
>         > 
>         > What do the others say to this? (Ideally ones that have already 
> used mspec or ones that understand the problem)
>         > 
>         > Chris
>         > 
>         > 
>         > Am 14.04.20, 15:46 schrieb "Łukasz Dywicki" <l...@code-house.org>:
>         > 
>         >     Hey Chris,
>         >     Sorry for wrongly addressed mail, it was intended to go to 
> mailing list.
>         >     
>         >     To the point - I think that what you propose with reordering 
> fields
>         >     manually shift the load from the tool to the developer. It will 
> lead to
>         >     further implementation errors caused by a need to reorder 
> fields while
>         >     creating spec. It will be inconsistent with rest of places 
> where things
>         >     are in "natural order" (big endian) except one or two types 
> which
>         >     collects bit flags which will have to be declared in different 
> order.
>         >     What I did in my naive implementation with "little endian" flag 
> on the
>         >     State type is re-arrangement of fields during generation time.
>         >     It does not require any byte shifting at the runtime and keeps 
> natural
>         >     order of fields without the need for manual reordering of flags 
> in mspec
>         >     declaration.
>         >     
>         >     I understand that keeping mspec simple is an important 
> priority, however
>         >     doing it at the cost of developers who will use it to implement 
> codecs
>         >     will not help us in long term.
>         >     I'm quite sure that sooner or later we will get a  protocol 
> with mix of
>         >     BE/LE fields.
>         >     
>         >     Cheers,
>         >     Łukasz
>         >     
>         >     
>         >     On 14.04.2020 14:57, Christofer Dutz wrote:
>         >     > Hi Lukasz (adding dev@ to the recipients again)
>         >     > 
>         >     >  
>         >     > 
>         >     > you are not quite correct. Enum types do declare a base type 
> where they
>         >     > are declared … not where they are used.
>         >     > 
>         >     >  
>         >     > 
>         >     > If you have bit-fields then why don’t you define them as 
> sequence of
>         >     > bits? I really can’t understand why you need to flip anything.
>         >     > 
>         >     > I think you’re trying to do something similar to Sebastian 
> when he
>         >     > wanted to introduce some bit-mask types.
>         >     > 
>         >     >  
>         >     > 
>         >     > I could imagine that in the spec it might be written … these 
> two bytes
>         >     > are a bit-mask and the bit 1 means X bit 10 means Y, …
>         >     > 
>         >     > So why not define the bit fields in the sequence they are 
> sent over the
>         >     > wire?
>         >     > 
>         >     >  
>         >     > 
>         >     > Assuming you have a bit field of 16 bits BE uint. If the spec 
> now says:
>         >     > 
>         >     >  
>         >     > 
>         >     >  
>         >     > 
>         >     > Bit:                        15|14|13|12|11|10|9 |8 |7 |6 |5  
> |4 |3  |2
>         >     >  |1  |0
>         >     > 
>         >     > Meaning:            A  |B  |C  |D  |E  | F  |G|H |I  | J | K 
> |L | M|N |O |P
>         >     > 
>         >     >  
>         >     > 
>         >     > Then the order they are sent is:
>         >     > 
>         >     > 7 |6 |5  |4 |3  |2  |1  |0 | 15|14|13|12|11|10|9 |8
>         >     > 
>         >     > So you have to declare the bit fields in the order:
>         >     > 
>         >     >                                I  | J | K |L | M|N |O |P  |A  
> |B  |C  |D
>         >     >  |E  | F  |G|H
>         >     > 
>         >     >  
>         >     > 
>         >     > So I don’t quite understand what you’re trying to flip here.
>         >     > 
>         >     >  
>         >     > 
>         >     > Chris
>         >     > 
>         >     >  
>         >     > 
>         >     > *Von: *Łukasz Dywicki <l...@code-house.org>
>         >     > *Datum: *Dienstag, 14. April 2020 um 13:26
>         >     > *An: *Christofer Dutz <christofer.d...@c-ware.de>
>         >     > *Betreff: *Re: Big and littleendian fields in one mspec
>         >     > 
>         >     >  
>         >     > 
>         >     > I've added the endianness switch to the types in my 
> experiments. These
>         >     > are not needed for enums as these are read as int and then 
> mapped to
>         >     > constants. I referred to these as an example - enum does have 
> a length
>         >     > indication and type behind it.
>         >     > 
>         >     >  
>         >     > 
>         >     > Again, I will bring the issue - StateIO currently fails to do 
> anything
>         >     > useful due to BE/LE switch. Because this type uses single 
> bits which
>         >     > need to be flipped before reading or after writing otherwise 
> they end up
>         >     > in improper order. Out of 16 bits 9 are in use so we have a 
> lot of
>         >     > possible combinations (2^9), which seems too much to create 
> an enum.
>         >     > 
>         >     > Since you didn't like my initial proposal and modifications 
> (as they
>         >     > might be redundant to what is available in the buffer API), 
> how would
>         >     > you handle State serialization without affecting mspec and 
> without
>         >     > further complication to code generation?
>         >     > 
>         >     >  
>         >     > 
>         >     > Best,
>         >     > 
>         >     > Łukasz
>         >     > 
>         >     >  
>         >     > 
>         >     > wt., 14 kwi 2020 o 12:47 Christofer Dutz 
> <christofer.d...@c-ware.de
>         >     > <mailto:christofer.d...@c-ware.de>> napisał(a):
>         >     > 
>         >     >     Hi Lukasz,
>         >     > 
>         >     >     but we don't have a:
>         >     >     [enum uint 16 little endian 'CommandId']
>         >     >     But only a:
>         >     >     [enum uint 16 'CommandId']
>         >     > 
>         >     >     And in your case I think perhaps the constants are not 
> correct. So
>         >     >     having a 16 bit uint will result in a 4 digit hex string:
>         >     >     So if you are having problems in mapping them to your 
> constants,
>         >     >     perhaps the constants are in the wrong endianness.
>         >     > 
>         >     >     I have encountered (but can't recall where) that the Enum 
> constants
>         >     >     in a BE protocol were written down in LE notation.
>         >     >     Of course the thing can't work then.
>         >     > 
>         >     >     So if for example you have the constant "1" in a BE 
> protocol with a
>         >     >     uint 16 enum type, your constant is not 0x0001, but 
> 0x0100 instead.
>         >     > 
>         >     >     Chris
>         >     > 
>         >     > 
>         >     > 
>         >     > 
>         >     >     Am 14.04.20, 12:37 schrieb "Łukasz Dywicki" 
> <l...@code-house.org
>         >     >     <mailto:l...@code-house.org>>:
>         >     > 
>         >     >         The legendary "two bytes" (shall we create a band 
> with this
>         >     >     name?!) are
>         >     >         coming from unfortunate State type. I don't mind 
> these to be an
>         >     >         int/sint/uint or whathever - this is matter of 
> interpretation.
>         >     > 
>         >     >         If you could please check again my earlier messages 
> you will find
>         >     >         struggle I have which is - how much given type takes.
>         >     > 
>         >     >         We have that for enums
>         >     >         [enum uint 16 little endian 'CommandId']
>         >     > 
>         >     >         but we don't have that for complex types
>         >     >         [type 'State']
>         >     > 
>         >     >         This leads to situation that we don't know how to 
> read and interpret
>         >     >         incoming byte sequence or how to properly write it 
> back to stream.
>         >     > 
>         >     >         It would be great if we could limit the issue just to
>         >     >         // read
>         >     >         state = StateIO.parseStatic(io.readUnsignedInt(16, 
> true)); //
>         >     >     true->LE
>         >     >         // write
>         >     >         wio = new WriteBuffer(2);
>         >     >         StateIO.staticSerialize(wio, new State(...));
>         >     >         io.writeUnsignedInt(8, wio.read, true); // true->LE
>         >     > 
>         >     >         However to do that first we need to know that State 
> is a uint 16
>         >     >     LE and
>         >     >         then do a lot of mambo-jambo in code generators to 
> cope with
>         >     >     that. :-)
>         >     > 
>         >     >         Best,
>         >     >         Łukasz
>         >     > 
>         >     > 
>         >     >         On 14.04.2020 12:20, Christofer Dutz wrote:
>         >     >         > Oh gee ... I totally remember having exactly the 
> same
>         >     >     discussion with Sebastian about "bytes" ...
>         >     >         >
>         >     >         > The problem is there is generally no "byte" "int" 
> and "uint"
>         >     >     because a byte = 8 bits is either signed or unsigned. 
> That should
>         >     >     the 3rd option be? Perhaps-signed?
>         >     >         > So generally I use "uint 8" for what you refer to 
> as a "byte 8".
>         >     >         >
>         >     >         > Wo what would be the difference between your "2 
> byte little
>         >     >     endian" and an "uint 16" with a LE ReadBuffer?
>         >     >         >
>         >     >         > I think what you have to rid yourself of thinking 
> of int as
>         >     >     number and a byte not being a number.
>         >     >         >
>         >     >         > Chris
>         >     >         >
>         >     >         >
>         >     >         >
>         >     >         > Am 14.04.20, 10:52 schrieb "Łukasz Dywicki"
>         >     >     <l...@code-house.org <mailto:l...@code-house.org>>:
>         >     >         >
>         >     >         >     Hey Christian,
>         >     >         >     The problem I face is quite simple. State type 
> in mspec i
>         >     >     declared as a
>         >     >         >     bunch of bits. Type length is fixed, however 
> not available
>         >     >     anywhere in
>         >     >         >     mspec or code generator to re-arrange bytes 
> upfront. All
>         >     >     we have exposed
>         >     >         >     at reader/write level is read/write bit.
>         >     >         >     To be fair LE/BE support in read/write buffers 
> is limited
>         >     >     just to
>         >     >         >     numbers. There is no such support for raw bytes 
> or bits,
>         >     >     cause for that
>         >     >         >     you need to declare a length of LE/BE sequence.
>         >     >         >     
>         >     >         >     I would love if I could just declare State as 
> '2 byte
>         >     >     little endian' so
>         >     >         >     it would be read properly upfront and parsed 
> with no
>         >     >     changes in
>         >     >         >     generated code, however I'm not sure how to do 
> it and
>         >     >     where. That's why
>         >     >         >     I'm playing with different things described in 
> earlier mail.
>         >     >         >     Since all type handling is general I am just 
> afraid of
>         >     >     more complicated
>         >     >         >     scenarios where we have variable length 
> structures such as
>         >     >     arrays.
>         >     >         >     
>         >     >         >     Best regards,
>         >     >         >     Łukasz
>         >     >         >     
>         >     >         >     
>         >     >         >     On 14.04.2020 09:41, Christofer Dutz wrote:
>         >     >         >     > Hi Lukasz,
>         >     >         >     >
>         >     >         >     > I am not sure I am understanding the problems 
> you are
>         >     >     facing. We already have LE and BE protocols.
>         >     >         >     > For example EIP is LE and the rest is 
> generally BE. It
>         >     >     seems that ADS/AMS is also LE.
>         >     >         >     > mspec doesn't even know about endianness.
>         >     >         >     >
>         >     >         >     > Up till now the endianness doesn't have an 
> effect on
>         >     >     bit-fields or single-bit ints.
>         >     >         >     > It only starts to affect if a field goes from 
> one byte
>         >     >     to the next, which is usually for (u)int and floating 
> point values.
>         >     >         >     >
>         >     >         >     > That's why we have created the 
> Read/WriteBuffers to set
>         >     >     their endianness in the constructor.
>         >     >         >     >
>         >     >         >     > So if you're creating a driver for ADS/AMS 
> which is LE,
>         >     >     then you write the mspec according to the sequence the 
> information
>         >     >     is located in the transferred bytes and have the 
> Read/WriteBuffer
>         >     >     handle the endianness issue.
>         >     >         >     >
>         >     >         >     > I do see a problem when there are drivers 
> that use mixed
>         >     >     endianness, but we have still to encounter such a 
> protocol.
>         >     >         >     >
>         >     >         >     > So I have to admit that I don't like any of 
> the mspec
>         >     >     changes you proposed, as I think you are just not using 
> the tools we
>         >     >     have the right way.
>         >     >         >     >
>         >     >         >     > Chris
>         >     >         >     >
>         >     >         >     >
>         >     >         >     >
>         >     >         >     > Am 14.04.20, 00:32 schrieb "Łukasz Dywicki"
>         >     >     <l...@code-house.org <mailto:l...@code-house.org>>:
>         >     >         >     >
>         >     >         >     >     Hey Niclas,
>         >     >         >     >     I realized how old the old things are 
> when I started
>         >     >     preparing
>         >     >         >     >     automation training for mere mortals and 
> got into
>         >     >     history of frames and
>         >     >         >     >     even cabling. Mr. Modbus and EIA-485 is 
> definitely
>         >     >     older than I. ;-)
>         >     >         >     >     
>         >     >         >     >     Getting back to the point - yes. I been 
> thinking how
>         >     >     to address the byte
>         >     >         >     >     order in effective way. Here are two 
> approaches I
>         >     >     have for now:
>         >     >         >     >     
>         >     >         >     >     A) My initial attempt is just a temporary 
> buffer
>         >     >     which is then written
>         >     >         >     >     in reverse order to caller. For reading 
> it is
>         >     >     similar - just getting N
>         >     >         >     >     bytes in reversed order. The hard part 
> is.. knowing
>         >     >     N. I had to add a
>         >     >         >     >     static calculation in order to allocate 
> valid buffer
>         >     >     sizes. I tend to
>         >     >         >     >     work but I'm not happy with this approach 
> cause it
>         >     >     involves additional work.
>         >     >         >     >     B) Second idea I've got is really simple 
> and relies
>         >     >     on code generation.
>         >     >         >     >     We know in which order fields are coming. 
> Here I'm
>         >     >     referring to a State
>         >     >         >     >     field which is just bunch of bits. If we 
> would group
>         >     >     fields in bytes and
>         >     >         >     >     generate code in reverse order then it 
> has chance to
>         >     >     work. Requirement
>         >     >         >     >     for that - ability to know basic field 
> sizes upfront.
>         >     >         >     >     C) Try to combine above with bit-io or
>         >     >     Read/WriteBuffers as these are
>         >     >         >     >     places which know actual position and 
> state of
>         >     >     buffers which are being
>         >     >         >     >     read/written.
>         >     >         >     >     
>         >     >         >     >     Now, getting to two cases which are a 
> problem.
>         >     >     CommandId and State. So
>         >     >         >     >     with command id situation is simple as it 
> is
>         >     >     declared as enum and it is
>         >     >         >     >     read as uint. We know size upfront and 
> can generate
>         >     >     valid method call
>         >     >         >     >     (readIntLE).
>         >     >         >     >     [enum uint 16 little endian 'CommandId'
>         >     >         >     >         ['0x00' INVALID]
>         >     >         >     >         ['0x01' ADS_READ_DEVICE_INFO]
>         >     >         >     >         ['0x02' ADS_READ]
>         >     >         >     >         ['0x03' ADS_WRITE]
>         >     >         >     >         ['0x04' ADS_READ_STATE]
>         >     >         >     >         ['0x05' ADS_WRITE_CONTROL]
>         >     >         >     >         ['0x06' ADS_ADD_DEVICE_NOTIFICATION]
>         >     >         >     >         ['0x07' 
> ADS_DELETE_DEVICE_NOTIFICATION]
>         >     >         >     >         ['0x08' ADS_DEVICE_NOTIFICATION]
>         >     >         >     >         ['0x09' ADS_READ_WRITE]
>         >     >         >     >     ]
>         >     >         >     >     
>         >     >         >     >     Second candidate is what I'm stuck right 
> now sniping
>         >     >     next cycles of
>         >     >         >     >     problems. So in case of State we have 
> complex type
>         >     >     composed from 2
>         >     >         >     >     bytes. A note here - instead of two bytes 
> we might
>         >     >     have a variable
>         >     >         >     >     length type which includes array or other 
> variable
>         >     >     section.
>         >     >         >     >     [type little endian 'State'
>         >     >         >     >         [simple     bit 'broadcast'           
>   ]
>         >     >         >     >         [reserved   int 7 '0x0'               
>   ]
>         >     >         >     >         [simple     bit 'initCommand'         
>   ]
>         >     >         >     >         [simple     bit 'updCommand'          
>   ]
>         >     >         >     >         [simple     bit 'timestampAdded'      
>   ]
>         >     >         >     >         [simple     bit 'highPriorityCommand' 
>   ]
>         >     >         >     >         [simple     bit 'systemCommand'       
>   ]
>         >     >         >     >         [simple     bit 'adsCommand'          
>   ]
>         >     >         >     >         [simple     bit 'noReturn'            
>   ]
>         >     >         >     >         [simple     bit 'response'            
>   ]
>         >     >         >     >     ]
>         >     >         >     >     
>         >     >         >     >     The order of reading big endian encoded 
> data to
>         >     >     impose little endian
>         >     >         >     >     shift would be (please correct me if I'm 
> wrong):
>         >     >         >     >     1) init
>         >     >         >     >     2) udp
>         >     >         >     >     3) add timestamp
>         >     >         >     >     4) priority
>         >     >         >     >     5) system
>         >     >         >     >     6) ads
>         >     >         >     >     7) noreturn
>         >     >         >     >     8) response (end of byte 1)
>         >     >         >     >     9) broadcast
>         >     >         >     >     10) reserved (end of byte )
>         >     >         >     >     We can do same trick for writing, by 
> re-arranging
>         >     >     fields. By this way we
>         >     >         >     >     avoid any additional byte level 
> operations.
>         >     >         >     >     
>         >     >         >     >     Overall trouble with generated driver is 
> to declare
>         >     >     "how much" bytes
>         >     >         >     >     should be read and interpreted. We have 
> precise size
>         >     >     information at the
>         >     >         >     >     runtime - due to length fields, we can 
> leverage it
>         >     >     at generation time,
>         >     >         >     >     but then we won't be able to cover all 
> cases.
>         >     >         >     >     
>         >     >         >     >     I would love to keep it simple and do not 
> break
>         >     >     things thus I need your
>         >     >         >     >     advice on how to approach this problem in 
> a valid way.
>         >     >         >     >     
>         >     >         >     >     Cheers,
>         >     >         >     >     Łukasz
>         >     >         >     >     
>         >     >         >     >     
>         >     >         >     >     On 13.04.2020 03:26, Niclas Hedhman wrote:
>         >     >         >     >     > <anecdotal-rant>
>         >     >         >     >     > For us who were around and shaping the 
> protocols
>         >     >     in the 1980s, and people
>         >     >         >     >     > before us (and before standards like 
> RS-232), a
>         >     >     lot of the "specifications"
>         >     >         >     >     > came out of "observation of 
> implementation we
>         >     >     managed to get to work",
>         >     >         >     >     > rather than "implement this spec". A 
> lot was due
>         >     >     to extreme memory
>         >     >         >     >     > constraints (in my case, multi-tasking 
> operating
>         >     >     system, serial protocol
>         >     >         >     >     > 187kbps, interpreted programming 
> language with
>         >     >     floating point ops and user
>         >     >         >     >     > applications in 2kB RAM and 8kB EPROM) 
> and a
>         >     >     general lack of information,
>         >     >         >     >     > like what other people were doing, 
> sharing
>         >     >     experiences and so on.
>         >     >         >     >     >
>         >     >         >     >     > And there were many "innovative" ways 
> to squeeze
>         >     >     just a little bit extra
>         >     >         >     >     > out of the hardware, resulting in "hard 
> to
>         >     >     understand" consequences. Bit
>         >     >         >     >     > packing was a typical one, multiple 
> functions
>         >     >     packed into a single byte.
>         >     >         >     >     > Look at page 14 in
>         >     >     https://www.nxp.com/docs/en/data-sheet/80C31_80C32.pdf
>         >     >         >     >     > and read up on "UART Enahanced Mode", 
> and we used
>         >     >     this, i.e. 9 bits, no
>         >     >         >     >     > parity and clever use of address and 
> mask to
>         >     >     create a slave-to-slave direct
>         >     >         >     >     > protocol, where the master's role was 
> to signal
>         >     >     which slave "owned" the
>         >     >         >     >     > cable. Yeah, in that 8kB ROM limitation 
> (I think
>         >     >     protocol was about 1kB
>         >     >         >     >     > ROM) and something like 150 bytes RAM 
> for comm
>         >     >     protocol.
>         >     >         >     >     >
>         >     >         >     >     > Could you implement a compatible device 
> to this
>         >     >     with PLC4X and modern
>         >     >         >     >     > hardware (i.e. no 8031/32 
> co-processor)? Possibly
>         >     >     but bit-banging is needed
>         >     >         >     >     > to support the 9bit data (+start and 
> stop bits)
>         >     >     and an awful lot of CPU
>         >     >         >     >     > cycles on something that was automatic 
> on one of
>         >     >     the slowest long-lived
>         >     >         >     >     > microcontroller ever.
>         >     >         >     >     > </anecdotal-rant>
>         >     >         >     >     >
>         >     >         >     >     > My point was only to highlight that 
> some of the
>         >     >     strange things you see in
>         >     >         >     >     > protocols today, have its roots in
>         >     >     pre-standardization days. Today no one
>         >     >         >     >     > would go down that route, because the 
> hardware
>         >     >     cost nothing now (8031  +
>         >     >         >     >     > 8kB EPROM + 2kB static RAM + battery 
> backup =>
>         >     >     ~$50 in 1983's currency) and
>         >     >         >     >     > longevity of software is more important.
>         >     >         >     >     >
>         >     >         >     >     > Cheers
>         >     >         >     >     > Niclas
>         >     >         >     >     >
>         >     >         >     >     >
>         >     >         >     >     > On Sun, Apr 12, 2020 at 10:10 PM 
> Christofer Dutz
>         >     >     <christofer.d...@c-ware.de 
> <mailto:christofer.d...@c-ware.de>>
>         >     >         >     >     > wrote:
>         >     >         >     >     >
>         >     >         >     >     >> Hi Lukasz,
>         >     >         >     >     >>
>         >     >         >     >     >> I think it really gets tricky when 
> using BE and
>         >     >     having some byte-odd-sizes
>         >     >         >     >     >> ... I remember in the Firmata protocol 
> there were
>         >     >     some bitmasks and then 10
>         >     >         >     >     >> bit uint as BE ... not it really got 
> tricky as
>         >     >     the specs were written from
>         >     >         >     >     >> a point of view: You read 16 bits BE 
> and then the
>         >     >     first6 bits mean XYZ
>         >     >         >     >     >> instead of describing how the bits 
> actually
>         >     >     travel over the wire.
>         >     >         >     >     >>
>         >     >         >     >     >> Chris
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >> Am 11.04.20, 01:21 schrieb "Łukasz 
> Dywicki"
>         >     >     <l...@code-house.org <mailto:l...@code-house.org>>:
>         >     >         >     >     >>
>         >     >         >     >     >>     I've made some progress with topic 
> by
>         >     >     modyfing mspec and allowing
>         >     >         >     >     >>     'little endian' flag on fields. 
> This moved me
>         >     >     further to next issue -
>         >     >         >     >     >>     which is whole type encoded little 
> endian.
>         >     >         >     >     >>
>         >     >         >     >     >>     In ADS driver such type is State, 
> which has 2
>         >     >     bytes and uses 8 bits for
>         >     >         >     >     >>     various flags.
>         >     >         >     >     >>     There are two cases which require 
> different
>         >     >     approach - reading and
>         >     >         >     >     >>     writing. So for reading we need to 
> swap N
>         >     >     bytes based on type length.
>         >     >         >     >     >>     For writing we need to alocate 
> buffer for N
>         >     >     bytes and swap them before
>         >     >         >     >     >>     writing.
>         >     >         >     >     >>
>         >     >         >     >     >>     I am stuck now with freemaker 
> templates and
>         >     >     bit-io.
>         >     >         >     >     >>
>         >     >         >     >     >>     Cheers,
>         >     >         >     >     >>     Łukasz
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >>     On 10.04.2020 17:57, Łukasz 
> Dywicki wrote:
>         >     >         >     >     >>     > I am doing some tests of ADS 
> serialization.
>         >     >         >     >     >>     >
>         >     >         >     >     >>     > I've run into some troubles with 
> payload
>         >     >     which is generated with new
>         >     >         >     >     >>     > driver. I'm not sure if that's 
> my fault or
>         >     >     generated code.
>         >     >         >     >     >>     >
>         >     >         >     >     >>     > I did a verification of what 
> Wireshark
>         >     >     shows and how ads structures
>         >     >         >     >     >> are
>         >     >         >     >     >>     > parsed. There is a gap I think. 
> For example
>         >     >     ams port number 1000
>         >     >         >     >     >>     > (0x1027) is read as 4135.
>         >     >         >     >     >>     >
>         >     >         >     >     >>     > Obviously I used wrong 
> structures while
>         >     >     implementing protocol logic
>         >     >         >     >     >> in
>         >     >         >     >     >>     > first place, but now I am 
> uncertain of how
>         >     >     fields are encoded. How we
>         >     >         >     >     >>     > mark field as little endian when 
> rest of
>         >     >     payload is big endian? Do we
>         >     >         >     >     >>     > have `uint_le`?
>         >     >         >     >     >>     >
>         >     >         >     >     >>     > As far I remember route creation 
> logic I
>         >     >     was tracking last week used
>         >     >         >     >     >>     > combination of LE and BE.
>         >     >         >     >     >>     >
>         >     >         >     >     >>     > Best regards,
>         >     >         >     >     >>     > Łukasz
>         >     >         >     >     >>     >
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >>
>         >     >         >     >     >
>         >     >         >     >     
>         >     >         >     >
>         >     >         >     
>         >     >         >
>         >     > 
>         >     
>         > 
> 
> 

Reply via email to