Hi Steve,
Now I've created a merged version of my 3 schemas in order to see if the
performance is better.
I did notice, that if I run tests that parsing usually takes about twice as
long in the merged schema.
The tests are running for inputs targeting the first 2 levels and I know that
now if parsing a level-2 input
Im parsing a TPKT packet with included COTP payload so I'm actually parsing two
levels, however if
I add the parsing time of TPKT and add that of simple COTP the sum is quite a
bit lower than that of
The combined schema. What could be causing this?
And I ran into some problems again :/ ... in my S7 Schema I have the case where
I need to output an (empty) payload element to match a parameter element.
Unfortunately doing this:
<xs:element name="S7RequestPayloadReadVar" type="xs:byte"
dfdl:lengthKind="explicit" dfdl:length="0"/>
Doesn't seem to work and I get the following error:
Expression Evaluation Error: Element s7f:S7RequestPayloadReadVar does not have
a value.
Schema context: element reference s7f:S7RequestPayloadReadVar Location line 423
column 46 in
file:/Users/christofer.dutz/Projects/Apache/PLC4X/protocols/target/classes/org/apache/plc4x/protocols/s7-full-stack-protocol.dfdl.xsd
How can I achieve this?
Chris
Am 22.01.19, 23:02 schrieb "Steve Lawrence" <[email protected]>:
If merged schemas allow you to access other fields to calculate the
length of the userData field instead of using delimited hexBinary, I
suspect you would see a noticeable performance increase.
Delimited hexBinary is implemented as encoding the input bytes into
ISO-8859-1 characters and building up a string until a delimiter or end
of data is found. The resulting string is then decoded to get the hex
binary byte array. It's not terribly slow, but is inefficient compared
to how we normally get hexBinary bytes with an explicit length. In the
explicit length case, we know exactly how many bits to read and can read
the source bytes directly into a hexBinary array, avoiding all the
encoding/decoding/delimiter scanning complexity.
- Steve
On 1/22/19 3:48 PM, Christofer Dutz wrote:
> Hi Steve
>
> Yup ... couldn't wait till tomorrow and yes ...
> your option worked (Wonder what I had different)
>
> Performance-wise ... would it be better to join the schemas?
>
> As I will always parse all 3 schemas and use them for serialization.
> I could imagine a merged schema (where I can for example get the
> length for COTP from the KPKT and use that for the userData)
>
> Chris
>
>
> Am 22.01.19, 18:44 schrieb "Steve Lawrence" <[email protected]>:
>
> Yep, I think hexBinay with dfdl:lengthKind="delimited" should work for
> your case. I've modified the userData element to look like this:
>
> <xs:element name="userData" type="xs:hexBinary"
> dfdl:byteOrder="bigEndian" dfdl:lengthKind="delimited"
> dfdl:encoding="ISO-8859-1" dfdl:textTrimKind="none" />
>
> This will cause the userData field to consume all data until the end
of
> the input. Note that delimited hexBinary is treated like string data,
so
> the encoding and textTrimKind properties need to be specified--it
might
> make sense to move them to the cotpFormat.
>
> I'm guessing the test you're talking about is "scenarioDataTpdu". With
> the above change to the schema and using the data from that test:
>
> 02F080320700000300000800080001120411440100ff09000401320004
>
> The resulting infoset is:
>
> <cotp:CoTpTPDU xmlns:cotp="http://plc4x.apache.org/cotp";>
> <headerLength>2</headerLength>
> <type>240</type>
> <cotp:CotpTpduData>
> <endOfTransmission>1</endOfTransmission>
> <tpduRef>0</tpduRef>
> </cotp:CotpTpduData>
>
<userData>320700000300000800080001120411440100FF09000401320004</userData>
> </cotp:CoTpTPDU>
>
> Three bytes total are consumed for the headerLength, type, and
> CotPTpduData field, and the remaining bytes end up in the userData
field
> as hexBinary. If there is no remaining data in the input, then the
> <userData> element is just empty (i.e. <userData />).
>
> - Steve
>
>
>
> On 1/22/19 11:58 AM, Christofer Dutz wrote:
> > Hi Steve,
> >
> > The code is in the plc4x repo I posted several times now.
Unfortunately I'm
> > sitting in a train without my laptop. It's the COTP protocol.
There's a matching
> > tdml test with commented out binary payload. That's what I'm trying
to read.
> >
> > Could probably post the links some time this evening.
> >
> > Chris
> >
> > Outlook für Android <https://aka.ms/ghei36> herunterladen
> >
> >
--------------------------------------------------------------------------------
> > *From:* Steve Lawrence <[email protected]>
> > *Sent:* Tuesday, January 22, 2019 5:17:24 PM
> > *To:* [email protected]; Christofer Dutz
> > *Subject:* Re: How to achieve lengthKind=”endOfParent” without
using endOfParent?
> > There isn't a concept of a global length of input since some inputs
> > could be streaming and so we don't actually know the length until
the
> > end of data is reached.
> >
> > I guess it isn't clear to me what your data looks like. I /think/
> > delimited hexBinary should work. If the parent element does not
have a
> > length, delimited hex binary should consume all available data up
until
> > the end. Could you provide a little more detail on what your data
looks
> > like (e.g. what has a known lengths, headers, user data, etc.)
> >
> > As far as implementing lengthKind="prefixed", I don't think the
current
> > Daffodil devs have the resources to implement endOfParent right now.
> > Most of us are focused on other tasks at the moment. Tough, it's
> > definitely possible to implement it--there aren't any real technical
> > limitations that I know of with the current code base--but it
probably
> > would be a decent amount of work and would be an ambitious tasks
for a
> > first time Daffodil contributor. Such a feature touches a lot of
> > different parts of Daffodil so there's a lot to learn. We're more
than
> > happy to provide guidance if you do want to contribute this
feature, and
> > it probably could be done in reasonably sized chunks, but I'd first
want
> > to confirm that there isn't an alternative.
> >
> > - Steve
> >
> >
> > On 1/22/19 10:35 AM, Christofer Dutz wrote:
> >> Hi Steve,
> >>
> >> well the problem is that I don't have the parent length in the
current context.
> >>
> >> Without it, it doesn't seem to work.
> >>
> >> If there was some sort of global variable providing the total
length of the entire input, that would be awesome.
> >> As I mentioned, the length information in in the surrounding
protocol, I wanted to model them all as separate as possible.
> >>
> >> Would it be possible to implement lengthKind="endOfParent"? Would
it be a lot of work? Could I help with it?
> >>
> >> Chris
> >>
> >>
> >>
> >> Am 22.01.19, 15:48 schrieb "Steve Lawrence" <[email protected]>:
> >>
> >> Correct, lengthKind="endOfParent" has not bee implemented yet.
> >>
> >> As an alternative that we do support, you should be able to use
> >> dfdl:lengthKind="delimited" for the hexBinary user data. In
this case,
> >> there's no delimiter, but parent length sort of acts like one.
For example:
> >>
> >> <xs:element name="Parent"
> >> dfdl:lengthKind="explicit" dfdl:length="4"
> >> dfdl:lengthUnits="bytes">
> >> <xs:complexType>
> >> <xs:sequence>
> >> <xs:element name="Header" type="xs:hexBinary"
> >> dfdl:lengthKind="explicit" dfdl:length="1"
> >> dfdl:lengthUnits="bytes" />
> >> <xs:element name="UserData" type="xs:hexBinary"
> >> dfdl:lengthKind="delimited"
dfdl:encoding="ISO-8859-1"/>
> >> </xs:sequence>
> >> </xs:complexType>
> >> </xs:element>
> >>
> >> So the parent element is 4 bytes and the header is 1 byte. If
we parse
> >> the data:
> >>
> >> 0xAA BB CC DD
> >>
> >> We get the following infoset
> >>
> >> <Parent>
> >> <Header>AA</Header>
> >> <UserData>BBCCDD</UserData>
> >> </Parent>
> >>
> >> And the UserData is the remaining three bytes. Using
> >> lengthKind="endOfParent" would probably have better
performance if we
> >> implemented it, but this should give the same result for the
hexBinary
> >> blob at the end.
> >>
> >> - Steve
> >>
> >>
> >> On 1/22/19 4:16 AM, Christofer Dutz wrote:
> >> > Hi all,
> >> >
> >> > I am stuck with a little problem … I am reading a packet,
which is usually contained inside another. Therefore it doesn’t provide any
means of providing it’s length.
> >> > So the packet is just a small header + binary data … now I
want to read “all the rest” after the header into a field “userData”.
> >> > In the DFDL documentation at IBM I could read that the
lengthKind=”endOfParent” would be what I’m looking for.
> >> >
> >> > Unfortunately this doesn’t seem to be supported … so how can
I achieve the same with implemented options?
> >> >
> >> > Chris
> >> >
> >>
> >>
> >>
> >
>
>
>
