Hey Chris,
Given the code generation is a serious piece of work and code to go with
I wanted to highlight just one thing. Lambda expressions might result in
messy stack traces.
Since frame parsing in some ways is recursive process it migth result in
multiple lambdas on stack trace. As long as we have conditionality only
with primitive fields it will work. Yet it will be much harder to debug
generated code if we will chain complex types (de)serialization.
I did update a while ago java ConversationContext to not rely on lambdas
so stack traces are always starting with org.apache.plc4x package. ;-)
All this should not hold you from doing your thing. I believe I will be
able to de-lambdize java templates if we will find it necessary later on.
Best,
Łukasz
On 07.10.2021 19:08, Christofer Dutz wrote:
Hi,
now that I'm sort of allmost finished with adjusting the mspec parsers and
model types, I'm planning up update the code generation templates.
I know that if I do it the normal way, that this would excessivley blow up the
complexity and size of the template files as well as that of the produced code.
As this code has become more and more complex over the last year anyway, I
would like to radically clean up the code here.
For that I had an idea, that I formalized a bit in the Confluence page:
https://cwiki.apache.org/confluence/display/PLC4X/MSPEC+improvements
TL/DR: I would create static methods that handle the reading/writing of every
field type and simply use these instead of generating code for every field.
Feel free to comment,
Chris
-----Ursprüngliche Nachricht-----
Von: Christofer Dutz <[email protected]>
Gesendet: Montag, 4. Oktober 2021 08:55
An: [email protected]
Betreff: AW: Changes on mspec: parameterized type refs, assert, try, const
Hi all,
so over the weekend Sebastian and I have been working hard on an updated mspec
version.
In the "feature/mspec-ng" is where all of this is happening.
So far the changes:
- We introduced a concept of "attributes" which can be added to fields and types (These
are name=expression) tuples. These are added before any argument-block. These can currently be used
for setting the encoding for string types (possibly even for other types in case we need them). And
for the upcomming "endianess" attribute which should allow switching endianess.
- We introduced a new simple type "vstring" for variable length strings. The
expression however is optional.
- The previous "string" now only accepts fixed number of bits.
- The encoding for String fields (string and vstring) defaults to "UTF-8" but you can
override it with an attribute with: encoding='"UTF-16"'
- The previous encoding string has been removed from string types)
- We added a new type of "field": "batchSet" which takes a list of attributes. These are
automatically added to all fields it contains. (Currently this is only available in "type",
"complexTyte" elements or inside typeSwitch case elements.
I think this was about it for now ... still need to implement the attribute
functionality ... it's currently parsed, but not processed yet.
Chris
-----Ursprüngliche Nachricht-----
Von: Christofer Dutz <[email protected]>
Gesendet: Dienstag, 28. September 2021 14:57
An: [email protected]
Betreff: AW: Changes on mspec: parameterized type refs, assert, try, const
Another thought I had last night ... while sort of not being able to sleep (at
least it hat something good)
So I think a "try" field should always be an optional field. As it might be set
or not, just with the difference that the condition is different.
This got me thinking even further ... how about making the condition of an
"optional" field optional? ... So in general what Sebastian was proposing with
a:
[try simple SomeType 'coolField']
Would actually be:
[optional SomeType 'coolField']
This would simplify things even more...
Chris
-----Ursprüngliche Nachricht-----
Von: Christofer Dutz <[email protected]>
Gesendet: Sonntag, 26. September 2021 16:03
An: [email protected]
Betreff: AW: Changes on mspec: parameterized type refs, assert, try, const
Bringing the thought of the other discussion here too ...
How about adding name-value paris to the type declarations as well as to the fields? Then the "try"
flag could be replaced with a name-value pair or even a name=expression pair ... sort of like
"parse-behaviour=try" or something similar. Then we wouldn't be starting to add all sorts of
"flags".
Chris
-----Ursprüngliche Nachricht-----
Von: Sebastian Rühl <[email protected]>
Gesendet: Sonntag, 26. September 2021 15:44
An: [email protected]
Betreff: Re: Changes on mspec: parameterized type refs, assert, try, const
Hey Lukaz,
the optional after the try is just a coincidence from the bac spec. Like it say
the low should always appear with a high and if the id is set then no name
should not be set. But in itself the try is self sufficient and behaves indeed
like an optional without an expression. Other than that you can always group
dependended fields as a complex... but let's see how this evolves.
These blocks could be indeed come in handy as an option to define things...
- Sebastian
On 2021/09/25 20:23:08, Łukasz Dywicki <[email protected]> wrote:
Hey Sebastian,
Not that far ago we had a similar discussion. Yet then it was mainly
about optionals. See "[DISCUSS] Change the way we use "optional".
Back then we did not do any changes to mspec, just ranted about
possible approach to make better use of optional fields.
I remember that current mspec/code can't for example handle ie. a
bacnet whois without all fields. Looking at your example I guess you
try to solve that puzzle.
I don't have any strong feelings here, but why not using optional
field with additional "reset" flag?
To me what rings a bell in your mail is an assumption that try is
always followed by optional field. This tells me that 'try' should
rather be a block than a field. By this way we will be able to pair
these visually as well as handle situation when there is one try
statement but multiple optional fields.
Best,
Łukasz
On 24.09.2021 23:52, Sebastian Rühl wrote:
Hi together,
I have some exciting changes in the pipeline regarding the mspec:
1. parameters on type refs
with that change it is now possible to target a discriminated child in
advance.
2. assert keyword
with that change it is possible to throw a
ParserAssertException (in java, or errors in other languages). This field is
similar to a const but instead of a ParseException a ParserAssertException is
thrown. In contrast to a const the check expression can be dynamic (e.g.
virtual fields now working on develop) 3. try keyword to prefix fields:
with that change it is possible to try to parse some content and in case
an assert fails it resets the buffer.
4. const is now extended to type reference
this change allows enums to be used as const values.
All theses changes allow to encapsulate behavior in complex types so you don't
need to DRY.
Here is a example working with bacnet:
['0x07' BACnetUnconfirmedServiceRequestWhoHas
[try simple BACnetComplexTagUnsignedInteger ['0',
'BACnetDataType.UNSIGNED_INTEGER' ] 'deviceInstanceRangeLowLimit' ]
[optional BACnetComplexTagUnsignedInteger ['1',
'BACnetDataType.UNSIGNED_INTEGER' ] 'deviceInstanceRangeHighLimit'
'deviceInstanceRangeLowLimit != null']
[try simple BACnetComplexTagOctetString ['2',
'BACnetDataType.OCTET_STRING' ] 'objectIdentifier' ]
[optional BACnetComplexTagOctetString ['3',
'BACnetDataType.OCTET_STRING' ] 'objectName' 'objectIdentifier == null' ]
]
The logic if a type matches is asserted in the type itself. The second optional
implies when the first element appears the second must be present. The last one
tries to read and if it fails it uses the second type.
Here is the snippet from the parent type:
[discriminatedType 'BACnetComplexTag' [uint 4 'tagNumberArgument',
BACnetDataType 'dataType']
[assert uint 4 'tagNumber'
'tagNumberArgument' ]
[const TagClass 'tagClass'
'TagClass.CONTEXT_SPECIFIC_TAGS' ]
[simple uint 3 'lengthValueType'
]
.....
[virtual uint 32 'actualLength' 'lengthValueType == 5
....']
[typeSwitch 'dataType'
....
['OCTET_STRING' BACnetComplexTagOctetString [uint 32 'actualLength']
// TODO: The reader expects int but uint32 get's mapped to long
so even uint32 would easily overflow...
[virtual uint 16
'actualLengthInBit' 'actualLength * 8']
[simple string 'actualLengthInBit' 'ASCII' 'theString']
]
Would love to hear some opinions! If there are no objections I would push this
change to develop soon.
- Sebastian
PatchContent:
Index:
code-generation/protocol-base-mspec/src/main/antlr4/org/apache/plc4x
/plugins/codegenerator/language/mspec/MSpec.g4
<+>UTF-8
===================================================================
diff --git
a/code-generation/protocol-base-mspec/src/main/antlr4/org/apache/plc4x/plugins/codegenerator/language/mspec/MSpec.g4
b/code-generation/protocol-base-mspec/src/main/antlr4/org/apache/plc4x/plugins/codegenerator/language/mspec/MSpec.g4
---
a/code-generation/protocol-base-mspec/src/main/antlr4/org/apache/plc4x/plugins/codegenerator/language/mspec/MSpec.g4
(revision ef35531d5a872f29dccddb3a11a135b166958185)
+++
b/code-generation/protocol-base-mspec/src/main/antlr4/org/apache/plc4x/plugins/codegenerator/language/mspec/MSpec.g4
(date 1632518519426)
@@ -34,7 +34,7 @@
;
fieldDefinition
- : LBRACKET field (LBRACKET params=multipleExpressions RBRACKET)?
RBRACKET
+ : LBRACKET tryParse? field (LBRACKET params=multipleExpressions
+ RBRACKET)? RBRACKET
;
dataIoDefinition
@@ -49,6 +49,7 @@
| discriminatorField
| enumField
| implicitField
+ | assertField
| manualArrayField
| manualField
| optionalField
@@ -73,7 +74,7 @@
;
constField
- : 'const' type=dataType name=idExpression expected=expression
+ : 'const' type=typeReference name=idExpression expected=expression
;
discriminatorField
@@ -88,6 +89,10 @@
: 'implicit' type=dataType name=idExpression serializeExpression=expression
;
+assertField
+ : 'assert' type=typeReference name=idExpression
+condition=expression ;
+
manualArrayField
: 'manualArray' type=typeReference name=idExpression
loopType=ARRAY_LOOP_TYPE loopExpression=expression parseExpression=expression
serializeExpression=expression lengthExpression=expression
;
@@ -129,7 +134,7 @@
;
typeReference
- : complexTypeReference=IDENTIFIER_LITERAL
+ : complexTypeReference=IDENTIFIER_LITERAL (LBRACKET
params=multipleExpressions RBRACKET)?
| simpleTypeReference=dataType
;
@@ -150,6 +155,10 @@
| base='dateTime'
;
+tryParse
+ : 'try'
+ ;
+
argument
: type=typeReference name=idExpression
;
