[jira] [Commented] (DAFFODIL-2200) xs:import problems with xsd files provided in daffodil-lib
[ https://issues.apache.org/jira/browse/DAFFODIL-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935336#comment-16935336 ] Michael Beckerle commented on DAFFODIL-2200: I have removed dchitlangia and added dineshchitlangia as contributor. > xs:import problems with xsd files provided in daffodil-lib > -- > > Key: DAFFODIL-2200 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2200 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Assignee: Dinesh Chitlangia >Priority: Major > Fix For: 2.5.0 > > > In dfdlx.xsd there are two xs:import statements. These are importing schemas > where the schemaLocation is the same directory where dfdlx.xsd resides. > Hence, the schemaLocation attributes should just be the filenames, not > "xsd/filename.xsd" as they are now. The "xsd/" prefix on the file names > should be removed. > In XMLSchema_for_DFDL.xsd, there is an import of the dfdl namespace. This has > no schemaLocation attribute. It should have > schemaLocation="DFDL_part3_model.xsd". > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (DAFFODIL-2200) xs:import problems with xsd files provided in daffodil-lib
[ https://issues.apache.org/jira/browse/DAFFODIL-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2200: -- Assignee: Dinesh Chitlangia (was: Dinesh Chitlangia) > xs:import problems with xsd files provided in daffodil-lib > -- > > Key: DAFFODIL-2200 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2200 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Assignee: Dinesh Chitlangia >Priority: Major > Fix For: 2.5.0 > > > In dfdlx.xsd there are two xs:import statements. These are importing schemas > where the schemaLocation is the same directory where dfdlx.xsd resides. > Hence, the schemaLocation attributes should just be the filenames, not > "xsd/filename.xsd" as they are now. The "xsd/" prefix on the file names > should be removed. > In XMLSchema_for_DFDL.xsd, there is an import of the dfdl namespace. This has > no schemaLocation attribute. It should have > schemaLocation="DFDL_part3_model.xsd". > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DAFFODIL-2206) user web site missing content about japanese MS-Windows language pack for utf-8
[ https://issues.apache.org/jira/browse/DAFFODIL-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934390#comment-16934390 ] Michael Beckerle commented on DAFFODIL-2206: The new page content should specifically call out the flaw of the windows command shell being non-UTF-8 capable, so that people are specifically forewarned about this. If there is some alternative command shell for windows (powershell?) that works better we should advise to use that. > user web site missing content about japanese MS-Windows language pack for > utf-8 > --- > > Key: DAFFODIL-2206 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2206 > Project: Daffodil > Issue Type: Bug > Components: Documentation >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > > The content from this page needs to be added to the user web site. > https://cwiki.apache.org/confluence/display/DAFFODIL/For+Contributors > This page exists on the wiki, but isn't the front page anymore, the content > seems unlinked. A search for "japanese" of the wiki finds it, but there's no > links to this. It belongs on the user wiki anyway. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DAFFODIL-2206) user web site missing content about japanese MS-Windows language pack for utf-8
Michael Beckerle created DAFFODIL-2206: -- Summary: user web site missing content about japanese MS-Windows language pack for utf-8 Key: DAFFODIL-2206 URL: https://issues.apache.org/jira/browse/DAFFODIL-2206 Project: Daffodil Issue Type: Bug Components: Documentation Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 The content from this page needs to be added to the user web site. https://cwiki.apache.org/confluence/display/DAFFODIL/For+Contributors This page exists on the wiki, but isn't the front page anymore, the content seems unlinked. A search for "japanese" of the wiki finds it, but there's no links to this. It belongs on the user wiki anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DAFFODIL-2204) unparserTestCase should dump hex data not just iso-8859-1 when mixed text/binary TDML doesn't compare
Michael Beckerle created DAFFODIL-2204: -- Summary: unparserTestCase should dump hex data not just iso-8859-1 when mixed text/binary TDML doesn't compare Key: DAFFODIL-2204 URL: https://issues.apache.org/jira/browse/DAFFODIL-2204 Project: Daffodil Issue Type: Improvement Components: TDML Runner Affects Versions: 2.4.0 Reporter: Michael Beckerle Assignee: Michael Beckerle Fix For: 2.5.0 Not enough info to figure out what's wrong. Need a hex dump as well as the quasi-text iso-8859-1 dump. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DAFFODIL-2203) Update project site, and Wiki, with info about our ASF Slack channel for support
Michael Beckerle created DAFFODIL-2203: -- Summary: Update project site, and Wiki, with info about our ASF Slack channel for support Key: DAFFODIL-2203 URL: https://issues.apache.org/jira/browse/DAFFODIL-2203 Project: Daffodil Issue Type: Improvement Components: Documentation, Website Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2202) Code Gen Framework
Michael Beckerle created DAFFODIL-2202: -- Summary: Code Gen Framework Key: DAFFODIL-2202 URL: https://issues.apache.org/jira/browse/DAFFODIL-2202 Project: Daffodil Issue Type: Improvement Components: Back End Affects Versions: 2.4.0 Reporter: Michael Beckerle Assignee: Julian Feinauer Fix For: 3.0.0 A backend that can generate code, rather than just execute scala code is needed. There are many people who want to have smaller-footprint lighter-weight implementations, but otherwise really like DFDL/Daffodil. Many people would contribute to fleshing this out if we put the framework in place to do it for a tiny subset of DFDL so that there is a pattern for people to follow. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (DAFFODIL-2081) TDML doc page description of twoPass incorrect
[ https://issues.apache.org/jira/browse/DAFFODIL-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2081: -- Assignee: Dinesh Chitlangia > TDML doc page description of twoPass incorrect > -- > > Key: DAFFODIL-2081 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2081 > Project: Daffodil > Issue Type: Bug > Components: Documentation, Website >Affects Versions: 2.3.0 >Reporter: Michael Beckerle >Assignee: Dinesh Chitlangia >Priority: Major > > The description of twoPass parserTestCases is not correct. > The first parse happens and creates infoset1. > The doc says infoset1 must not match the expected infoset, but that's not > correct. It may or may not match. Often it will match. > Then the unparse happens and creates outData1 > OutData1 must not match the original input data. If it did a onePass test > would be sufficient. > So that part is correct. > Then the second parse happens and creates infoset2. > Infoset2 must match the expected infoset. > This can be verified by editing the CSV example. Add a second separator by > changing dfdl:separator="," to dfdl:separator="| ," so that pipe or comma are > accepted as separators, but pipe will be output by the unparser since it is > listed first. > Parser test cases that are default onePass will now fail, because the unparse > produces data with pipes, but the input data had commas. > However, the infoset created by the first parse does match what is expected. > Changing these tests to "twoPass" makes them work. That means the fact that > the first infoset comparison matched the expected is tolerated. The doc says > it is not. > Note that besides the doc page, the tdml.xsd in daffodil-lib has > documentation in comments in it, and that is also wrong. > > > > > > > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (DAFFODIL-2200) xs:import problems with xsd files provided in daffodil-lib
[ https://issues.apache.org/jira/browse/DAFFODIL-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2200: -- Assignee: Dinesh Chitlangia > xs:import problems with xsd files provided in daffodil-lib > -- > > Key: DAFFODIL-2200 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2200 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Assignee: Dinesh Chitlangia >Priority: Major > Fix For: 2.5.0 > > > In dfdlx.xsd there are two xs:import statements. These are importing schemas > where the schemaLocation is the same directory where dfdlx.xsd resides. > Hence, the schemaLocation attributes should just be the filenames, not > "xsd/filename.xsd" as they are now. The "xsd/" prefix on the file names > should be removed. > In XMLSchema_for_DFDL.xsd, there is an import of the dfdl namespace. This has > no schemaLocation attribute. It should have > schemaLocation="DFDL_part3_model.xsd". > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2201) Test Daffodil using GraalVM
Michael Beckerle created DAFFODIL-2201: -- Summary: Test Daffodil using GraalVM Key: DAFFODIL-2201 URL: https://issues.apache.org/jira/browse/DAFFODIL-2201 Project: Daffodil Issue Type: Improvement Components: CLI Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 GraalVM is openjdk based. It *should* be able to run Daffodil as it can run Scala code. It features mechanism to output native binaries which massively increases startup speed. It supposedly also makes scalac and other scala things run faster than they do on ordinary openjdk. We should evaluate it. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2200) xs:import problems with xsd files provided in daffodil-lib
Michael Beckerle created DAFFODIL-2200: -- Summary: xs:import problems with xsd files provided in daffodil-lib Key: DAFFODIL-2200 URL: https://issues.apache.org/jira/browse/DAFFODIL-2200 Project: Daffodil Issue Type: Bug Components: Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 In dfdlx.xsd there are two xs:import statements. These are importing schemas where the schemaLocation is the same directory where dfdlx.xsd resides. Hence, the schemaLocation attributes should just be the filenames, not "xsd/filename.xsd" as they are now. The "xsd/" prefix on the file names should be removed. In XMLSchema_for_DFDL.xsd, there is an import of the dfdl namespace. This has no schemaLocation attribute. It should have schemaLocation="DFDL_part3_model.xsd". -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2199) initiatedContent="yes" does not cause SDE when initiator="%ES;"
Michael Beckerle created DAFFODIL-2199:
--
Summary: initiatedContent="yes" does not cause SDE when
initiator="%ES;"
Key: DAFFODIL-2199
URL: https://issues.apache.org/jira/browse/DAFFODIL-2199
Project: Daffodil
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Michael Beckerle
Fix For: 2.5.0
This schema fragment does not cause an SDE as required by the DFDL spec:
{code:java}
{code}
{color:#003296}Note that the DFDL spec says that when initiatedContent is yes,
the initiator may not be empty string, but this is ambiguous. Clearly
dfdl:initiator="" is not allowed, but a check for dfdl:initiator="%ES;" should
also be rejected. {color}
{color:#003296}Actually, dfdl:initiator="%WSP*;" should also be rejected in
this case.{color}
{color:#003296}dfdl:initiator="%ES;" should uniformly, always be rejected,
because it is meaningless in all cases. {color}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (DAFFODIL-2197) abort: There are no references to this component
[ https://issues.apache.org/jira/browse/DAFFODIL-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-2197. Resolution: Fixed > abort: There are no references to this component > > > Key: DAFFODIL-2197 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2197 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.5.0 >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > > An abort occurs with message "There are no references to this component" for > schemas (e.g., VMF) > This should either be a warning, or should just be silently ignored. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-2197) abort: There are no references to this component
[ https://issues.apache.org/jira/browse/DAFFODIL-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924590#comment-16924590 ] Michael Beckerle commented on DAFFODIL-2197: In review: https://github.com/apache/incubator-daffodil/pull/267 > abort: There are no references to this component > > > Key: DAFFODIL-2197 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2197 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.5.0 >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > > An abort occurs with message "There are no references to this component" for > schemas (e.g., VMF) > This should either be a warning, or should just be silently ignored. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (DAFFODIL-2197) abort: There are no references to this component
[ https://issues.apache.org/jira/browse/DAFFODIL-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2197: -- Assignee: Michael Beckerle > abort: There are no references to this component > > > Key: DAFFODIL-2197 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2197 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.5.0 >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > > An abort occurs with message "There are no references to this component" for > schemas (e.g., VMF) > This should either be a warning, or should just be silently ignored. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2197) abort: There are no references to this component
Michael Beckerle created DAFFODIL-2197: -- Summary: abort: There are no references to this component Key: DAFFODIL-2197 URL: https://issues.apache.org/jira/browse/DAFFODIL-2197 Project: Daffodil Issue Type: Bug Components: Front End Affects Versions: 2.5.0 Reporter: Michael Beckerle Fix For: 2.5.0 An abort occurs with message "There are no references to this component" for schemas (e.g., VMF) This should either be a warning, or should just be silently ignored. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2196) XML Schema for DFDL is invalid
Michael Beckerle created DAFFODIL-2196: -- Summary: XML Schema for DFDL is invalid Key: DAFFODIL-2196 URL: https://issues.apache.org/jira/browse/DAFFODIL-2196 Project: Daffodil Issue Type: Bug Components: Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 The XML Schema for DFDL Schemas contains a broken, circular, type definition for boolean. !https://attachments.office.net/owa/mbecke...@tresys.com/service.svc/s/GetAttachmentThumbnail?id=AAMkAGNlM2NiMDA1LTBmMjQtNDExOS05OThkLTBiZTM2YjJhMjhlMABGAAC7M%2FjUiiZrQaUtdTD88ILjBwCGZaTuXXzVTrtqA%2F1w5gdwAAEJAACGZaTuXXzVTrtqA%2F1w5gdwAAHI9WcQAAABEgAQANpwxh6K5RBArbW1PlOIGF4%3D&thumbnailType=2&owa=outlook.office.com&scriptVer=2019082310.17&X-OWA-CANARY=jwuKIWr7cEq-TNVEVczMbwAF6NXyMtcYG-XfhIdJq-djxCpkRojfaKw5Wtd6gqYdn56KGdoIco8.&token=eyJhbGciOiJSUzI1NiIsImtpZCI6IjA2MDBGOUY2NzQ2MjA3MzdFNzM0MDRFMjg3QzQ1QTgxOENCN0NFQjgiLCJ4NXQiOiJCZ0Q1OW5SaUJ6Zm5OQVRpaDhSYWdZeTN6cmciLCJ0eXAiOiJKV1QifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsInZlciI6IkV4Y2hhbmdlLkNhbGxiYWNrLlYxIiwiYXBwY3R4c2VuZGVyIjoiT3dhRG93bmxvYWRAYTBkNDU2NjctNmMwNy00ZTg4LTg2OGYtNGFjOWFmOTVjN2VkIiwiYXBwY3R4Ijoie1wibXNleGNocHJvdFwiOlwib3dhXCIsXCJwcmltYXJ5c2lkXCI6XCJTLTEtNS0yMS0yOTI1MzkwNzU2LTQxODU2NjM4Mi04NDM4MzgwNjktNjU4NDIxOVwiLFwicHVpZFwiOlwiMTE1MzgzNjI5Njc1NDkxMjYwOVwiLFwib2lkXCI6XCJiZjlmZThmOC02YzQ1LTRmOWQtOWJhNS02M2NjZDRiMmYzOThcIixcInNjb3BlXCI6XCJPd2FEb3dubG9hZFwifSIsIm5iZiI6MTU2Nzc5MjA2NCwiZXhwIjoxNTY3NzkyNjY0LCJpc3MiOiIwMDAwMDAwMi0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDBAYTBkNDU2NjctNmMwNy00ZTg4LTg2OGYtNGFjOWFmOTVjN2VkIiwiYXVkIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwL2F0dGFjaG1lbnRzLm9mZmljZS5uZXRAYTBkNDU2NjctNmMwNy00ZTg4LTg2OGYtNGFjOWFmOTVjN2VkIn0.pRYQAQpsE2QaOZMY76T6iLSuj4SYEQAMjfGcSJa5Pi8TtcQAh7RFYW8iMuXURAJrLSwMoRYvgLfKkTTXJdH8Ba-kCuTXmFrinGPRLSh7I104-nTB2ak1mStNoA_I3Wx_Bu-exz62Mx516vh9EmXEVGiel0VAFhp6g_aIJM3SYOBnOBoscuXxhRaHW2_j6Iah309FsgGvN_orBlSX3SiBsukFvTPKC-3j2X8kYu19uQRmKbXOEMDGfZqyzxg6kPdyAtkFm2uVmMa36LlehyCNS00IBx6zfX2LzebQ2JSRTIjRMxLdbiJ17TXXpbqFcI_I180J3XHc5t7QQ6ucvV8SaQ&animation=true! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DAFFODIL-2195) Diagnostic file URLs are misleading when saved in compiled schema
Michael Beckerle created DAFFODIL-2195: -- Summary: Diagnostic file URLs are misleading when saved in compiled schema Key: DAFFODIL-2195 URL: https://issues.apache.org/jira/browse/DAFFODIL-2195 Project: Daffodil Issue Type: Bug Components: Diagnostics Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 URLs stored for diagnostic purposes in compiled schemas contain absolute file names present in the compilation environment. This is problematic because it leaks information such as the userID of whomever built it. These URLs should be shortened to just be relative URLs and exclude a prefix of the directory structure so that part that is provided is meaningful and useful, but does not include home directory information of a user. See related DAFFODIL-1200 and DAFFODIL-2072. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-1200) Really long path names in diagnostic messages make them hard to read
[ https://issues.apache.org/jira/browse/DAFFODIL-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922603#comment-16922603 ] Michael Beckerle commented on DAFFODIL-1200: These long URLs also create possible false-positive situations where a negative test looking for a keyword in the diagnostic message may match part of a directory or file name in the URL, even when the expected message is NOT received. > Really long path names in diagnostic messages make them hard to read > > > Key: DAFFODIL-1200 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1200 > Project: Daffodil > Issue Type: Bug > Components: Diagnostics, Usability >Affects Versions: s15 >Reporter: Michael Beckerle >Priority: Minor > > Giant long file paths like > file:/home/mbeckerle/dataiti/git/daffodil/eclipse-projects/target/test/edu/illinois/ncsa/daffodil/section06/namespaces.tdml > These clutter the error messages, and all but the section06/namespaces.tdml > are noise. > Almost nothing can be viewed without scrolling the window/panel to the right > so as to get past these giant paths. Or alternatively they wrap around line > after line. > Any given path being displayed can have whatever the longest prefix of it > that matches a location on the classpath, that longest prefix can be removed > without _much_ loss of information content. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (DAFFODIL-1444) Performance - schema compilation
[ https://issues.apache.org/jira/browse/DAFFODIL-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-1444: -- Assignee: (was: Michael Beckerle) > Performance - schema compilation > > > Key: DAFFODIL-1444 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1444 > Project: Daffodil > Issue Type: Improvement > Components: Front End, Middle "End", Performance >Reporter: Michael Beckerle >Priority: Major > Attachments: Daffodil-exponential-component-growth.xlsx > > > Large DFDL schemas are very slow to compile. > We could focus on speeding this up, and should get some low-hanging fruit > here. > But ultimately, a really large DFDL schema needs to be compiled in pieces. > (DEBATABLE - focus should FIRST be on speeding up and reducing the massive > copying that goes on. Separate compilation is a harder issue that we can > defer.) > This means we need to be able to reload a compiled schema just to restore > it's parsers/unparsers and associated runtime data structures to memory so > that another schema that depends on it can then be compiled. > DFDL schema compilation needs to be understood in order to decompose a schema > into separately compilable units. THere's no point in trying to compile a > schema layer by layer - a DFDL schema containing all type definitions, for > example, doesn't compile to anything. There have to be top level elements in > order for DFDL schema compilation to do anything. > So given a large data format with many top-level element types, we need the > compiler to recognize element references to pre-compiled top-level elements, > and avoid recompiling new instances of them if the surrounding environment is > the same. That is, surrounding default format specification is the same. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-2194) buffered data output stream has a chunk limit of 2GB
[ https://issues.apache.org/jira/browse/DAFFODIL-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918014#comment-16918014 ] Michael Beckerle commented on DAFFODIL-2194: Agree. I recall that the Microsoft BizTalk parser had a tuning parameter around amount of lookahead, which could be set even to zero which would prevent any PoUs. Another way for the cache-limit to work is that if the cache grows sufficiently past a limit, it serves as a discriminator and all enclosing PoU are marked as discriminated (literally run up the stack of them, setting them all to true), at which point a bunch of buffers can be freed. A failure at that point would then be fatal as there would be no PoU to backtrack to. But a parse could succeed so long as no backtracking was actually needed. One challenge with the no-PoU-with-BLOB thing is that it has bad composition properties. Modifying a schema to replace a hexBinary with a BLOB might cause the need to rethink the whole way the schema works to avoid the enclosing PoUs. > buffered data output stream has a chunk limit of 2GB > > > Key: DAFFODIL-2194 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2194 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.5.0 > > > A buffered data outupt stream is backed by a growable ByteArrayOutputStream, > which can only grow to 2GB in size. So if we ever try to write more than 2GB > to a buffered output stream during unparse (very possible with large blobs), > we'll get an OutOfMemoryError. > One potential solution is to be aware of the size of a ByteArrayOutputStream > when buffering output and automatically create a split when it gets to 2GB in > sizes. This will still require a ton of memory since we're buffering these in > memoary, but we'll at least be able to unparse more than 2GB of continuous > data. > Note that we should still be able to unparse more than 2GB of data total, as > long as there so single buffer that's more than 2GB. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-2194) buffered data output stream has a chunk limit of 2GB
[ https://issues.apache.org/jira/browse/DAFFODIL-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917998#comment-16917998 ] Michael Beckerle commented on DAFFODIL-2194: Good points. I think one thing to consider is perhaps we don't allow points of uncertainty around blobs at all. I.e., you must have occursCountKind="explicit" so there are no optional element PoUs, and we require choices to be choice-by-dispatch. Then there can be no backtracking. It would be an SDE to have a blob inside of a PoU. Note that a choice with PoU with multiple branches the BLOB could still be in the last branch, as by then there is no more backtracking. A BLOB could also appear in a speculative branch if *after* a discriminator. There may be some other caveats, but it might be possible to just say PoU and BLOB don't mix. > buffered data output stream has a chunk limit of 2GB > > > Key: DAFFODIL-2194 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2194 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.5.0 > > > A buffered data outupt stream is backed by a growable ByteArrayOutputStream, > which can only grow to 2GB in size. So if we ever try to write more than 2GB > to a buffered output stream during unparse (very possible with large blobs), > we'll get an OutOfMemoryError. > One potential solution is to be aware of the size of a ByteArrayOutputStream > when buffering output and automatically create a split when it gets to 2GB in > sizes. This will still require a ton of memory since we're buffering these in > memoary, but we'll at least be able to unparse more than 2GB of continuous > data. > Note that we should still be able to unparse more than 2GB of data total, as > long as there so single buffer that's more than 2GB. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-2194) buffered data output stream has a chunk limit of 2GB
[ https://issues.apache.org/jira/browse/DAFFODIL-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917969#comment-16917969 ] Michael Beckerle commented on DAFFODIL-2194: True enough. If I just have a protocol buffer that works like HTTP, the entire thing, even though it is just an array of lines of text, must be measured for length because the header has to carry the total length. So even without any blobs, a 2.1GByte page of small objects would blow this limit up even if the ONLY suspension was for this single header length field. So this fix is really two fixes. 1 to fix 2GByte + boundary on output buffer sizes, and the other to avoid bringing blobs into memory at all. > buffered data output stream has a chunk limit of 2GB > > > Key: DAFFODIL-2194 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2194 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.5.0 > > > A buffered data outupt stream is backed by a growable ByteArrayOutputStream, > which can only grow to 2GB in size. So if we ever try to write more than 2GB > to a buffered output stream during unparse (very possible with large blobs), > we'll get an OutOfMemoryError. > One potential solution is to be aware of the size of a ByteArrayOutputStream > when buffering output and automatically create a split when it gets to 2GB in > sizes. This will still require a ton of memory since we're buffering these in > memoary, but we'll at least be able to unparse more than 2GB of continuous > data. > Note that we should still be able to unparse more than 2GB of data total, as > long as there so single buffer that's more than 2GB. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DAFFODIL-2194) buffered data output stream has a chunk limit of 2GB
[ https://issues.apache.org/jira/browse/DAFFODIL-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917916#comment-16917916 ] Michael Beckerle commented on DAFFODIL-2194: I don't think fixing this by just auto-splitting DirectOrBufferedDataOutputStream if buffering and the buffer grows near 2GByte max is a good fix. It will work, and allow larger blobs to be processed, but doesn't keep the blob data out of the java heap. We will be able to get past the 2GByte object size limit, but the java heap size limit will be next. We want to be able to parse image/video files that are much larger than the java heap would want to grow to. I.e., a 64GByte NITF file parsed in a 5GByte JVM. That means that the DirectOrBufferedDataOutputStream would need a specific blob-indirect flavor. So when you unparse a blob, rather than stream it into a buffered instance, copying all the bytes into one or more java heap objects of size <= 2GB, you instead just defer the blob and point at the file (and add in its length to the accumulated length for use by the dfdl:valueLength function). When the streams collapse together that's when the blob file contents would be accessed and streamed to the direct output stream. > buffered data output stream has a chunk limit of 2GB > > > Key: DAFFODIL-2194 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2194 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.5.0 > > > A buffered data outupt stream is backed by a growable ByteArrayOutputStream, > which can only grow to 2GB in size. So if we ever try to write more than 2GB > to a buffered output stream during unparse (very possible with large blobs), > we'll get an OutOfMemoryError. > One potential solution is to be aware of the size of a ByteArrayOutputStream > when buffering output and automatically create a split when it gets to 2GB in > sizes. This will still require a ton of memory since we're buffering these in > memoary, but we'll at least be able to unparse more than 2GB of continuous > data. > Note that we should still be able to unparse more than 2GB of data total, as > long as there so single buffer that's more than 2GB. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (DAFFODIL-2185) Add more charset encoding for obscure 2, 3, 4, 5 bit encodings
[ https://issues.apache.org/jira/browse/DAFFODIL-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-2185. Resolution: Fixed Fixed in d3e8f82cebad4a6dea51c75759a5ece0a09fc289 > Add more charset encoding for obscure 2, 3, 4, 5 bit encodings > -- > > Key: DAFFODIL-2185 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2185 > Project: Daffodil > Issue Type: New Feature > Components: Back End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > > A variety of formats require some odd character encodings. These need to be > added. > A 2-bit encoding which can produce character 0, 1, 2, 3 is needed. > A 3-bit octal variant which roduces 1 to 8 (not 0 to 7) > And several others are needed for data formats such as link16. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2191) Warning about varying alignment is sporadic.
[ https://issues.apache.org/jira/browse/DAFFODIL-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900290#comment-16900290 ] Michael Beckerle commented on DAFFODIL-2191: Perhaps this is due to the confusing isOptional and isArray predicates inside Daffodil. This error check is probably calling isOptional thinking it means "might not be present", when the definition of isOptional is actually "has 0 or 1 occurrence". I think isVariableOccurrences is likely the right test here. > Warning about varying alignment is sporadic. > - > > Key: DAFFODIL-2191 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2191 > Project: Daffodil > Issue Type: Bug > Components: Diagnostics, Middle "End" >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.5.0 > > Attachments: binSchema.dfdl.xsd, compErr.tdml > > > Diagnostic warning about alignment is issued for attached schema and TDML, > but if you remove the maxOccurs="unbounded" from Section 3, then there is no > warning. > So we get the warning for dfdl:occursCountKind="implicit" when > maxOccurs="unbounded" but not when bounded. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2191) Warning about varying alignment is sporadic.
Michael Beckerle created DAFFODIL-2191: -- Summary: Warning about varying alignment is sporadic. Key: DAFFODIL-2191 URL: https://issues.apache.org/jira/browse/DAFFODIL-2191 Project: Daffodil Issue Type: Bug Components: Diagnostics, Middle "End" Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 Attachments: binSchema.dfdl.xsd, compErr.tdml Diagnostic warning about alignment is issued for attached schema and TDML, but if you remove the maxOccurs="unbounded" from Section 3, then there is no warning. So we get the warning for dfdl:occursCountKind="implicit" when maxOccurs="unbounded" but not when bounded. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2190) Poor diagnostic - identifier of schema component having error uses "group[4]"
[
https://issues.apache.org/jira/browse/DAFFODIL-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2190:
---
Labels: beginner (was: )
> Poor diagnostic - identifier of schema component having error uses "group[4]"
> -
>
> Key: DAFFODIL-2190
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2190
> Project: Daffodil
> Issue Type: Bug
> Components: Diagnostics
>Reporter: Michael Beckerle
>Priority: Major
> Labels: beginner
>
>
> {code:java}
> [warning] Schema Definition Warning: Section_3 is an optional element or a
> variable-occurrence array and its alignment (1) is not the same as group[4]'s
> alignment (8).{code}
>
> {color:#33}In the schema associated with this message, there is only one
> group definition, and the group reference referring to it is the 4th child of
> the enclosing element, but it is a xs:sequence with
> dfdl:hiddenGroupRef.{color}
> {color:#33}Referring to this as "group[4]" is not helpful. Some sort of
> schema component ID, or an XPath that identifies the component at issue
> usefully is required. {color}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2190) Poor diagnostic - identifier of schema component having error uses "group[4]"
[
https://issues.apache.org/jira/browse/DAFFODIL-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2190:
---
Component/s: Diagnostics
> Poor diagnostic - identifier of schema component having error uses "group[4]"
> -
>
> Key: DAFFODIL-2190
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2190
> Project: Daffodil
> Issue Type: Bug
> Components: Diagnostics
>Reporter: Michael Beckerle
>Priority: Major
>
>
> {code:java}
> [warning] Schema Definition Warning: Section_3 is an optional element or a
> variable-occurrence array and its alignment (1) is not the same as group[4]'s
> alignment (8).{code}
>
> {color:#33}In the schema associated with this message, there is only one
> group definition, and the group reference referring to it is the 4th child of
> the enclosing element, but it is a xs:sequence with
> dfdl:hiddenGroupRef.{color}
> {color:#33}Referring to this as "group[4]" is not helpful. Some sort of
> schema component ID, or an XPath that identifies the component at issue
> usefully is required. {color}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2190) Poor diagnostic - identifier of schema component having error uses "group[4]"
[
https://issues.apache.org/jira/browse/DAFFODIL-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2190:
---
Description:
{code:java}
[warning] Schema Definition Warning: Section_3 is an optional element or a
variable-occurrence array and its alignment (1) is not the same as group[4]'s
alignment (8).{code}
{color:#33}In the schema associated with this message, there is only one
group definition, and the group reference referring to it is the 4th child of
the enclosing element, but it is a xs:sequence with dfdl:hiddenGroupRef.{color}
{color:#33}Referring to this as "group[4]" is not helpful. Some sort of
schema component ID, or an XPath that identifies the component at issue
usefully is required. {color}
> Poor diagnostic - identifier of schema component having error uses "group[4]"
> -
>
> Key: DAFFODIL-2190
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2190
> Project: Daffodil
> Issue Type: Bug
>Reporter: Michael Beckerle
>Priority: Major
>
>
> {code:java}
> [warning] Schema Definition Warning: Section_3 is an optional element or a
> variable-occurrence array and its alignment (1) is not the same as group[4]'s
> alignment (8).{code}
>
> {color:#33}In the schema associated with this message, there is only one
> group definition, and the group reference referring to it is the 4th child of
> the enclosing element, but it is a xs:sequence with
> dfdl:hiddenGroupRef.{color}
> {color:#33}Referring to this as "group[4]" is not helpful. Some sort of
> schema component ID, or an XPath that identifies the component at issue
> usefully is required. {color}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2190) Poor diagnostic - identifier of schema component having error uses "group[4]"
[ https://issues.apache.org/jira/browse/DAFFODIL-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2190: --- Summary: Poor diagnostic - identifier of schema component having error uses "group[4]" (was: Poor diagnostic - identifier of schema component having error uses "group[4]) > Poor diagnostic - identifier of schema component having error uses "group[4]" > - > > Key: DAFFODIL-2190 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2190 > Project: Daffodil > Issue Type: Bug >Reporter: Michael Beckerle >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2190) Poor diagnostic - identifier of schema component having error uses "group[4]
Michael Beckerle created DAFFODIL-2190: -- Summary: Poor diagnostic - identifier of schema component having error uses "group[4] Key: DAFFODIL-2190 URL: https://issues.apache.org/jira/browse/DAFFODIL-2190 Project: Daffodil Issue Type: Bug Reporter: Michael Beckerle -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2189) Unparse with trace on causes Abort
[
https://issues.apache.org/jira/browse/DAFFODIL-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894083#comment-16894083
]
Michael Beckerle commented on DAFFODIL-2189:
We need some unit tests that run with trace on, so as to insure that the
debugger/trace works for a round-trip test. Otherwise it is too easy to break
something that only fails under tracing/debug, but not notice.
> Unparse with trace on causes Abort
> --
>
> Key: DAFFODIL-2189
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2189
> Project: Daffodil
> Issue Type: Bug
> Components: Debugger
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Major
> Fix For: 2.5.0
>
>
> I tried to trace a round-trip test. The parse side works. The unparse does
> not.
> {code:java}
> org.apache.daffodil.exceptions.Abort: Invariant broken:
> org.apache.daffodil.util.Maybe.WithNulls.isDefined[org.apache.daffodil.infoset.DINode](UState.this.currentInfosetNode)
> org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
> org.apache.daffodil.processors.unparsers.UState.infoset(UState.scala:124)
> org.apache.daffodil.debugger.InteractiveDebugger$DebugCommandBase$Info$InfoInfoset$.act(InteractiveDebugger.scala:1506)
> org.apache.daffodil.debugger.InteractiveDebugger$DebugCommandBase$Info$.$anonfun$act$20(InteractiveDebugger.scala:1209)
> at org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
> at
> org.apache.daffodil.processors.unparsers.UState.infoset(UState.scala:124)
> at
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2189) Unparse with trace on causes Abort
Michael Beckerle created DAFFODIL-2189:
--
Summary: Unparse with trace on causes Abort
Key: DAFFODIL-2189
URL: https://issues.apache.org/jira/browse/DAFFODIL-2189
Project: Daffodil
Issue Type: Bug
Components: Debugger
Affects Versions: 2.4.0
Reporter: Michael Beckerle
Fix For: 2.5.0
I tried to trace a round-trip test. The parse side works. The unparse does not.
{code:java}
org.apache.daffodil.exceptions.Abort: Invariant broken:
org.apache.daffodil.util.Maybe.WithNulls.isDefined[org.apache.daffodil.infoset.DINode](UState.this.currentInfosetNode)
org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
org.apache.daffodil.processors.unparsers.UState.infoset(UState.scala:124)
org.apache.daffodil.debugger.InteractiveDebugger$DebugCommandBase$Info$InfoInfoset$.act(InteractiveDebugger.scala:1506)
org.apache.daffodil.debugger.InteractiveDebugger$DebugCommandBase$Info$.$anonfun$act$20(InteractiveDebugger.scala:1209)
at org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
at org.apache.daffodil.processors.unparsers.UState.infoset(UState.scala:124)
at
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2188) inputTypeCalc sometimes fails due to BigInt vs. JBigInt
Michael Beckerle created DAFFODIL-2188: -- Summary: inputTypeCalc sometimes fails due to BigInt vs. JBigInt Key: DAFFODIL-2188 URL: https://issues.apache.org/jira/browse/DAFFODIL-2188 Project: Daffodil Issue Type: Bug Components: Back End, Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 In working with DFDL schemas that use the inputTypeCalc function, it is sometimes passed a java.math.BigInteger (aka JBigInt by our conventions), when what is stored in the hash-table it uses are scala.math.BigInt objects. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2187) too many debug/oolagdebug messages from sbt it:test
Michael Beckerle created DAFFODIL-2187: -- Summary: too many debug/oolagdebug messages from sbt it:test Key: DAFFODIL-2187 URL: https://issues.apache.org/jira/browse/DAFFODIL-2187 Project: Daffodil Issue Type: Bug Components: Infrastructure, QA Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 The volume of output created by sbt it:test is so large that when something fails in a travis CI build, it is almost impossible to see what went wrong. Most of this output is created by tests that turn on various levels of logging that really aren't relevant any more. Tests that turn on debug or oolagdebug level of detail should be revisited to see whether they should still be run with those settings off, or just disabled. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-788) add lines-of-code counting as an sbt target
[ https://issues.apache.org/jira/browse/DAFFODIL-788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890484#comment-16890484 ] Michael Beckerle commented on DAFFODIL-788: --- I did substantial work on this, but it is parked on github on OpenDFDL at g...@github.com:OpenDFDL/sbt-stats.git > add lines-of-code counting as an sbt target > --- > > Key: DAFFODIL-788 > URL: https://issues.apache.org/jira/browse/DAFFODIL-788 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Affects Versions: s10 >Reporter: Michael Beckerle >Priority: Minor > > LineCounter2 class is in daffodil-core under src/test/scala. > It would be good if this was just driven directly from sbt so that it could > be a standard automated part of a built/test/release cycle. > It should be modified to not require a hard-wired path name, etc. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2186) User defined functions for DPath expressions
Michael Beckerle created DAFFODIL-2186: -- Summary: User defined functions for DPath expressions Key: DAFFODIL-2186 URL: https://issues.apache.org/jira/browse/DAFFODIL-2186 Project: Daffodil Issue Type: New Feature Components: Back End, Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 Some things cannot be reasonably expressed in DFDL's DPath expression language. A good example of this is the abililty to convert between elevation measured in height above geoid (aka elevation above mean sea level or MSL), and height-above-elipsoid or HAE. This requires a table of numbers and interpolation. Nevertheless a DFDL schema needs to convert units of measure and it is very convenient to have the output of a DFDL parse to be normalized into a desired unit of measure. The ability to define a function in a separate jar library, and call it from a DPath expression, having registered it somehow so Daffodil can find it, is what I'm calling a user-defined function. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DAFFODIL-2168) Provide support for Non-spacing mark/Combining Characters in Debugger
[
https://issues.apache.org/jira/browse/DAFFODIL-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle resolved DAFFODIL-2168.
Resolution: Fixed
Fix Version/s: 2.5.0
Fixed in 8555e25b14555b0ff4f271ea1170083a8d6c1a13
> Provide support for Non-spacing mark/Combining Characters in Debugger
> -
>
> Key: DAFFODIL-2168
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2168
> Project: Daffodil
> Issue Type: Bug
> Components: Debugger
>Affects Versions: 2.4.0
>Reporter: Olabusayo Kilo
>Assignee: Olabusayo Kilo
>Priority: Major
> Fix For: 2.5.0
>
> Attachments: nonspacingmark.png, tcp.dfdl.xsd, test.dump
>
>
> Certain unicode characters are [combining
> characters|https://en.wikipedia.org/wiki/Combining_character] and can combine
> with the characters surrounding them, modifying them. This can lead to
> misalignment in our debugger.
> A possible solution could be to detect when a character is a non-spacing
> (or other combining character) and insert the appropriate number of spaces
> around it. Java's Character.getType method will return an enum about what
> type a character is. Many of these character types might need this alignment
> assistance.
> Attached is a pictorial example of the problem and the files that can be used
> to recreate the issue.
> Command run:
> {code:java}
> daffodil --trace parse -Dtransport:Protocol=6 transport:DataLength=79 -s
> tcp.dfdl.xsd test.dump{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2172) Choice keys are using scala.math.BigInt instead of java.lang.BigInteger
[ https://issues.apache.org/jira/browse/DAFFODIL-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2172: --- Component/s: Back End > Choice keys are using scala.math.BigInt instead of java.lang.BigInteger > --- > > Key: DAFFODIL-2172 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2172 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Brandon Sloane >Assignee: Brandon Sloane >Priority: Major > > This seems mostly related to some of the recent typeCalc stuff, but Scala's > BigInt is sneaking into the system and causing issues. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-1625) SAPI and JAPI Diagnostic getSomeCause returns Throwable - should be a Daffodil exception.
[ https://issues.apache.org/jira/browse/DAFFODIL-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1625: --- Component/s: Clean Ups > SAPI and JAPI Diagnostic getSomeCause returns Throwable - should be a > Daffodil exception. > - > > Key: DAFFODIL-1625 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1625 > Project: Daffodil > Issue Type: Improvement > Components: API, Clean Ups >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Priority: Major > > Developer at BBN was integrating Daffodil via JAPI, was surprised that the > ParseResult getDiagnostics() method returns a list of Diagnostic, and those > have getSomeCause() which returns a Throwable. > They expected this to be an encpasulating Daffodil-specific exception wrapper > around whatever the raw throwable is. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-1515) dfdl:checkRangeInclusive and dfdl:checkRangeExclusive functions
[ https://issues.apache.org/jira/browse/DAFFODIL-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1515: --- Labels: beginner (was: ) > dfdl:checkRangeInclusive and dfdl:checkRangeExclusive functions > --- > > Key: DAFFODIL-1515 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1515 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language, Front End >Reporter: Michael Beckerle >Priority: Minor > Labels: beginner > > See https://redmine.ogf.org/issues/314 > New DFDL functions: > dfdl:checkRangeInclusive($node, $val1, $val2) > dfdl:checkRangeExclusive($node, $val1, $val2) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-1114) DPath: fn:namespace-uri() Unsupported Function
[
https://issues.apache.org/jira/browse/DAFFODIL-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-1114:
---
Labels: beginner (was: )
> DPath: fn:namespace-uri() Unsupported Function
> --
>
> Key: DAFFODIL-1114
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1114
> Project: Daffodil
> Issue Type: New Feature
> Components: DFDL Language
>Affects Versions: s15
>Reporter: Elizabeth Fahl
>Priority: Minor
> Labels: beginner
>
> When running the recently-created fn:namespace-uri tests against the
> dpath-with-serialization2 branch, all tests fail with the following:
> [error] Schema Definition Error: Unsupported function:
> fn:namespace-uri{xmlns:fn='http://www.w3.org/2005/xpath-functions'}
> [error] Schema context: element.e3. Location line 39 column 88 in
> file:/home/efinnegan/Projects/daffodil/daffodil-test/target/scala-2.10/test-classes/edu/illinois/ncsa/daffodil/section23/dfdl_functions/home_schema.dfdl.xsd.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2185) Add more charset encoding for obscure 2, 3, 4, 5 bit encodings
Michael Beckerle created DAFFODIL-2185: -- Summary: Add more charset encoding for obscure 2, 3, 4, 5 bit encodings Key: DAFFODIL-2185 URL: https://issues.apache.org/jira/browse/DAFFODIL-2185 Project: Daffodil Issue Type: New Feature Components: Back End Affects Versions: 2.4.0 Reporter: Michael Beckerle Assignee: Michael Beckerle Fix For: 2.5.0 A variety of formats require some odd character encodings. These need to be added. A 2-bit encoding which can produce character 0, 1, 2, 3 is needed. A 3-bit octal variant which roduces 1 to 8 (not 0 to 7) And several others are needed for data formats such as link16. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-451) Implement a Check for the Distinctness of Delimiters
[ https://issues.apache.org/jira/browse/DAFFODIL-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-451: -- Issue Type: Improvement (was: New Feature) > Implement a Check for the Distinctness of Delimiters > > > Key: DAFFODIL-451 > URL: https://issues.apache.org/jira/browse/DAFFODIL-451 > Project: Daffodil > Issue Type: Improvement > Components: Middle "End", Usability >Reporter: Jessie Chab >Priority: Minor > Time Spent: 49m > > Taylor: Do you know if the padChar is allowed to be the same as an escapeChar? > Mike: bad idea. I think we added an errata just a week or so ago that says it > is an SDE if any of the things are not distinct. E.g., the terminator cannot > be the same as the pad character either, nor any other delimiter, nor the > textStandardGroupingSeparator nor \ > Mike: delimiters can be shared prefixes of each other, and an enclosing group > could have the same delimiters as something it encloses (e.g., enclosing > gorup has comma separator, enclosed inner group does also) We do need an > explicit check for the distinctness of these things. So jessie's test should > throw an SDE. > So... that test should result in an SDE. We'll need to create a ticket/bug to > add that. > Conversation above first referenced in DFDL-259 (Implement Trim/Pad/Fill) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-1565) Unparse API Cursor-behavior/streaming enhancements
[ https://issues.apache.org/jira/browse/DAFFODIL-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890325#comment-16890325 ] Michael Beckerle commented on DAFFODIL-1565: changed title to reflect that parse aspect is complete. Related or duplicate of DAFFODIL-1526 ? > Unparse API Cursor-behavior/streaming enhancements > -- > > Key: DAFFODIL-1565 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1565 > Project: Daffodil > Issue Type: Improvement > Components: API, Performance >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > > See review comment for context. > A thought. This API wants to eventually be something where we can call it > over and over, picking up where the prior unparse left off in the output. > That might be a different API that is slightly different, but in general we > need to be able to leave the output in a state such that the next unparse > call can continue where the previous left off. Doing this at the > bit-position granularity might be too hard, but byte-position ought to be > possible. > That's not really about this API, which is symmetric with our parse API. It's > just an additional requirement. (The parse API also needs this > call-over-and-over capability.) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-1565) Unparse API Cursor-behavior/streaming enhancements
[ https://issues.apache.org/jira/browse/DAFFODIL-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1565: --- Summary: Unparse API Cursor-behavior/streaming enhancements (was: Parse/Unparse API Cursor-behavior/streaming enhancements) > Unparse API Cursor-behavior/streaming enhancements > -- > > Key: DAFFODIL-1565 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1565 > Project: Daffodil > Issue Type: Improvement > Components: API, Performance >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > > See review comment for context. > A thought. This API wants to eventually be something where we can call it > over and over, picking up where the prior unparse left off in the output. > That might be a different API that is slightly different, but in general we > need to be able to leave the output in a state such that the next unparse > call can continue where the previous left off. Doing this at the > bit-position granularity might be too hard, but byte-position ought to be > possible. > That's not really about this API, which is symmetric with our parse API. It's > just an additional requirement. (The parse API also needs this > call-over-and-over capability.) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-1539) scrutinize use of ULong.longValue
[ https://issues.apache.org/jira/browse/DAFFODIL-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890323#comment-16890323 ] Michael Beckerle commented on DAFFODIL-1539: I assigned to Brandon as he's looking at the AnyRef issue, and this might be related. Unassign please if this proves to be orthogonal or separate fix. > scrutinize use of ULong.longValue > - > > Key: DAFFODIL-1539 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1539 > Project: Daffodil > Issue Type: Improvement > Components: Back End, Clean Ups >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Brandon Sloane >Priority: Major > > We use ULong in two different ways (possibly more, but this is about these > two) > We use them as values coming out of data parsing or going into data > unparsing. In this case, all 64 bits are used, and we don't care if the > underlying stored Long integer is negative or positive. > We also use them as bit offsets/positions into the data stream. This assures > that they are always non-negative when created. > However, the math that computes bit offsets calls ULong.longValue a great > deal, then operates on the Long elements. > This is technically incorrect. Now, one can argue that it won't matter > because you need an offset bigger than 2^63 ... before you get a ULong where > the underlying Long would be negative. > But, this is the stuff of security holes. E.g., a data format can store a > length that results in a bit length > 2^63. No such data can exist, but some > prefix header field can be created that suggests that much data. > And then who is to say that something won't break? > So, the right thing here is for arithmetic on ULong to work directly (I'm not > sure why it doesn't. There appear to be operators defined.) Or for a method > other than ULong.longValue be used when what we are assuming we're getting > back is a non-negative Long. > We should scrutinize all calls to ULong.longValue and see if they can/should > be changed to ULong.asPosition or ULong.asLength, etc. methods that express > the intent, or the conversion can just be avoided and math done directly on > ULong values. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (DAFFODIL-1539) scrutinize use of ULong.longValue
[ https://issues.apache.org/jira/browse/DAFFODIL-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-1539: -- Assignee: Brandon Sloane > scrutinize use of ULong.longValue > - > > Key: DAFFODIL-1539 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1539 > Project: Daffodil > Issue Type: Improvement > Components: Back End, Clean Ups >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Brandon Sloane >Priority: Major > > We use ULong in two different ways (possibly more, but this is about these > two) > We use them as values coming out of data parsing or going into data > unparsing. In this case, all 64 bits are used, and we don't care if the > underlying stored Long integer is negative or positive. > We also use them as bit offsets/positions into the data stream. This assures > that they are always non-negative when created. > However, the math that computes bit offsets calls ULong.longValue a great > deal, then operates on the Long elements. > This is technically incorrect. Now, one can argue that it won't matter > because you need an offset bigger than 2^63 ... before you get a ULong where > the underlying Long would be negative. > But, this is the stuff of security holes. E.g., a data format can store a > length that results in a bit length > 2^63. No such data can exist, but some > prefix header field can be created that suggests that much data. > And then who is to say that something won't break? > So, the right thing here is for arithmetic on ULong to work directly (I'm not > sure why it doesn't. There appear to be operators defined.) Or for a method > other than ULong.longValue be used when what we are assuming we're getting > back is a non-negative Long. > We should scrutinize all calls to ULong.longValue and see if they can/should > be changed to ULong.asPosition or ULong.asLength, etc. methods that express > the intent, or the conversion can just be avoided and math done directly on > ULong values. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2169) Add type safety to DPath variables
[ https://issues.apache.org/jira/browse/DAFFODIL-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890320#comment-16890320 ] Michael Beckerle commented on DAFFODIL-2169: In principle one can define mathematical operations on these AnyVal types, so that one can perform various operations on them without extracting an AnyRef object first. This might actually help make the diagnostic messaging more uniform. > Add type safety to DPath variables > -- > > Key: DAFFODIL-2169 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2169 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups >Reporter: Brandon Sloane >Assignee: Brandon Sloane >Priority: Major > > Currently our DPath runtime (and, I believe, main infoset) pass around values > of type AnyRef, which affords almost no protection through the Scala type > system. > This was done to avoid the boxing overhead normaly associated with creating > case classes. However, it should be possible to create an unboxed type that > can provide some level of type safety. > Simmilar to our current Maybe type, can can create a new value type that > wraps AnyRef. Being a value type, it will not require boxing. We can then > make the main constructor hidden, and provide constructors only for those > types that we want to allow as DPath variables. > This should provide us type safty in assiging to DPath variables. Reading the > values will still involve dealing with AnyRef however. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DAFFODIL-1782) DFDL Extension: Enum and Range table lookups
[ https://issues.apache.org/jira/browse/DAFFODIL-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-1782. Resolution: Fixed Fix Version/s: 2.4.0 Implemented as an experimental feature (dfdlx). > DFDL Extension: Enum and Range table lookups > > > Key: DAFFODIL-1782 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1782 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language, Front End >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > Some schemas have large simple type unions that combine enumerations with > many members, and numeric ranges. > Translation between numeric values, and symbolic values (strings), is > desirable. > A proposal for how to do this is: > [https://cwiki.apache.org/confluence/display/DAFFODIL/Proposal%3A+Features+to+Support+Table-Lookup] > (This one is now obsolete: > [https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Enumerations+and+Range+Tables+via+Simple+Type+Unions]) > �� > Under review by the DFDL Workgroup, it was observed that these features > require validation processing, and that was considered problematic given that > validation is an optional feature of DFDL.�� However, the > dfdl:checkConstraints function is not optional - excepting that the whole > expression language is optional. So it is reasonable to add these features to > an implementation that has either validation or dfdl:checkConstraints(). > �� -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (DAFFODIL-639) unicodeByteOrderMark feature
[ https://issues.apache.org/jira/browse/DAFFODIL-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle closed DAFFODIL-639. - > unicodeByteOrderMark feature > > > Key: DAFFODIL-639 > URL: https://issues.apache.org/jira/browse/DAFFODIL-639 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language >Reporter: Michael Beckerle >Priority: Minor > > This is not a property. The unicodeByteOrderMark is a member of the Infoset > Document Item. (aka the root element). > It depends on the dfdl:encoding property, which can be a runtime expression; > hence, this must be computed in an Evaluatable which in turn evaluates the > encodingEv. > Likely an Evaluatable[Option[ByteOrder]] is the type. > If no encoding property is defined this should be a constant None. > If the encoding property is defined and known to NOT be one of UTF-8, UTF-16, > or UTF-32, then this should be a constant None. > When unparsing, the value will either have been set from parsing, or can be > set from an API call. (New API method on Infoset needed.) > The API call is allowed, but the value ignored/unused by the unparser unless > the encoding is UTF-8, UTF-16, or UTF-32. > When the encoding evaluates to UTF-8, then the unicodeByteOrderMark will be > determined by the first 3 bytes being: > * 0xEF 0xBF 0xBE - ByteOrder.LittleEndian - 3 bytes are consumed (note: > strictly speaking, this shouldn't occur, but will if a naive utf-8 encoder > encodes a little-endian BOM into a 3-byte UTF-8 sequence. To insure such data > will round trip between UTF-8 and UTF-16 (LE - via BOM), we match this > sequence, and choose LittleEndian byte order) > * 0xEF 0xBB 0xBF - ByteOrder-BigEndian - 3 bytes are consumed > * anything else - no bytes are consumed, and the unicodeByteOrderMark is not > set (has no value) > when unparsing, if unicodeByteOrderMark is not set, then no byte order mark > is output. > For UTF-16, > * 0xFE 0xFF - byteOrder.BigEndian - 2 bytes are consumed > * 0xFF 0xFE - byteOrder.LittleEndian - 2 bytes are consumed > * anything else - parse error. > When unparsing, if encoding is UTF-16, and unicodeByteOrderMark is not set - > unparse error. > UTF-32 works like utf-16, except the byte patterns are 00 00 FE FF for > bigEndian, and FF FE 00 00 for littleEndian. > Recommended: package this code for reuse, assuming it needs to be used as a > library for reading/decoding strings generally. It's not impossible that the > above runtime errors when the byte order is not known, will be augmented in > the future by a mode where each individual text string at fine granularity is > examined for a byte order mark at the start. There also may be a need for > utf-16 heuristic byte-order determination - that is by looking at the bytes > for the characters and determining if they make more sense as big-endian or > little endian. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DAFFODIL-639) unicodeByteOrderMark feature
[ https://issues.apache.org/jira/browse/DAFFODIL-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-639. --- Resolution: Won't Fix > unicodeByteOrderMark feature > > > Key: DAFFODIL-639 > URL: https://issues.apache.org/jira/browse/DAFFODIL-639 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language >Reporter: Michael Beckerle >Priority: Minor > > This is not a property. The unicodeByteOrderMark is a member of the Infoset > Document Item. (aka the root element). > It depends on the dfdl:encoding property, which can be a runtime expression; > hence, this must be computed in an Evaluatable which in turn evaluates the > encodingEv. > Likely an Evaluatable[Option[ByteOrder]] is the type. > If no encoding property is defined this should be a constant None. > If the encoding property is defined and known to NOT be one of UTF-8, UTF-16, > or UTF-32, then this should be a constant None. > When unparsing, the value will either have been set from parsing, or can be > set from an API call. (New API method on Infoset needed.) > The API call is allowed, but the value ignored/unused by the unparser unless > the encoding is UTF-8, UTF-16, or UTF-32. > When the encoding evaluates to UTF-8, then the unicodeByteOrderMark will be > determined by the first 3 bytes being: > * 0xEF 0xBF 0xBE - ByteOrder.LittleEndian - 3 bytes are consumed (note: > strictly speaking, this shouldn't occur, but will if a naive utf-8 encoder > encodes a little-endian BOM into a 3-byte UTF-8 sequence. To insure such data > will round trip between UTF-8 and UTF-16 (LE - via BOM), we match this > sequence, and choose LittleEndian byte order) > * 0xEF 0xBB 0xBF - ByteOrder-BigEndian - 3 bytes are consumed > * anything else - no bytes are consumed, and the unicodeByteOrderMark is not > set (has no value) > when unparsing, if unicodeByteOrderMark is not set, then no byte order mark > is output. > For UTF-16, > * 0xFE 0xFF - byteOrder.BigEndian - 2 bytes are consumed > * 0xFF 0xFE - byteOrder.LittleEndian - 2 bytes are consumed > * anything else - parse error. > When unparsing, if encoding is UTF-16, and unicodeByteOrderMark is not set - > unparse error. > UTF-32 works like utf-16, except the byte patterns are 00 00 FE FF for > bigEndian, and FF FE 00 00 for littleEndian. > Recommended: package this code for reuse, assuming it needs to be used as a > library for reading/decoding strings generally. It's not impossible that the > above runtime errors when the byte order is not known, will be augmented in > the future by a mode where each individual text string at fine granularity is > examined for a byte order mark at the start. There also may be a need for > utf-16 heuristic byte-order determination - that is by looking at the bytes > for the characters and determining if they make more sense as big-endian or > little endian. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-639) unicodeByteOrderMark feature
[ https://issues.apache.org/jira/browse/DAFFODIL-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890316#comment-16890316 ] Michael Beckerle commented on DAFFODIL-639: --- This feature is going to be removed from the DFDL v1.0 spec. So closing this ticket. > unicodeByteOrderMark feature > > > Key: DAFFODIL-639 > URL: https://issues.apache.org/jira/browse/DAFFODIL-639 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language >Reporter: Michael Beckerle >Priority: Minor > > This is not a property. The unicodeByteOrderMark is a member of the Infoset > Document Item. (aka the root element). > It depends on the dfdl:encoding property, which can be a runtime expression; > hence, this must be computed in an Evaluatable which in turn evaluates the > encodingEv. > Likely an Evaluatable[Option[ByteOrder]] is the type. > If no encoding property is defined this should be a constant None. > If the encoding property is defined and known to NOT be one of UTF-8, UTF-16, > or UTF-32, then this should be a constant None. > When unparsing, the value will either have been set from parsing, or can be > set from an API call. (New API method on Infoset needed.) > The API call is allowed, but the value ignored/unused by the unparser unless > the encoding is UTF-8, UTF-16, or UTF-32. > When the encoding evaluates to UTF-8, then the unicodeByteOrderMark will be > determined by the first 3 bytes being: > * 0xEF 0xBF 0xBE - ByteOrder.LittleEndian - 3 bytes are consumed (note: > strictly speaking, this shouldn't occur, but will if a naive utf-8 encoder > encodes a little-endian BOM into a 3-byte UTF-8 sequence. To insure such data > will round trip between UTF-8 and UTF-16 (LE - via BOM), we match this > sequence, and choose LittleEndian byte order) > * 0xEF 0xBB 0xBF - ByteOrder-BigEndian - 3 bytes are consumed > * anything else - no bytes are consumed, and the unicodeByteOrderMark is not > set (has no value) > when unparsing, if unicodeByteOrderMark is not set, then no byte order mark > is output. > For UTF-16, > * 0xFE 0xFF - byteOrder.BigEndian - 2 bytes are consumed > * 0xFF 0xFE - byteOrder.LittleEndian - 2 bytes are consumed > * anything else - parse error. > When unparsing, if encoding is UTF-16, and unicodeByteOrderMark is not set - > unparse error. > UTF-32 works like utf-16, except the byte patterns are 00 00 FE FF for > bigEndian, and FF FE 00 00 for littleEndian. > Recommended: package this code for reuse, assuming it needs to be used as a > library for reading/decoding strings generally. It's not impossible that the > above runtime errors when the byte order is not known, will be augmented in > the future by a mode where each individual text string at fine granularity is > examined for a byte order mark at the start. There also may be a need for > utf-16 heuristic byte-order determination - that is by looking at the bytes > for the characters and determining if they make more sense as big-endian or > little endian. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle resolved DAFFODIL-2184.
Resolution: Not A Bug
Fix Version/s: 2.5.0
Closing. User was not using a released version of Daffodil. Using 2.4.0 solved
it.
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Major
> Fix For: 2.5.0
>
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Closed] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle closed DAFFODIL-2184.
--
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Major
> Fix For: 2.5.0
>
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887181#comment-16887181
]
Michael Beckerle commented on DAFFODIL-2184:
User reports they were using 2.3.0-SNAPSHOT not 2.4.0 as they expected. Using
the right version solves the issue.
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Major
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2184:
---
Priority: Major (was: Critical)
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Major
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887161#comment-16887161
]
Michael Beckerle commented on DAFFODIL-2184:
Unable to reproduce with test materials sent by user.
However those differ a bit from what created the above error message (Label vs.
HiddenLabel), so possibly need more accurate example.
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Critical
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
[
https://issues.apache.org/jira/browse/DAFFODIL-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2184:
---
Priority: Critical (was: Major)
> No line number in dfdl:assert parse error message.
> --
>
> Key: DAFFODIL-2184
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Critical
>
> user reports this diagnostic, which in prior revisions of Daffodil would have
> had a line number in it.
>
> {code:java}
> [error] Parse Error: Assertion failed: { . eq 9 } failed
> Schema context: HiddenLabel Location in test.dfdl.xsd
> Data location was preceding byte 1
> {code}
>
> this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2184) No line number in dfdl:assert parse error message.
Michael Beckerle created DAFFODIL-2184:
--
Summary: No line number in dfdl:assert parse error message.
Key: DAFFODIL-2184
URL: https://issues.apache.org/jira/browse/DAFFODIL-2184
Project: Daffodil
Issue Type: Bug
Components: Back End
Affects Versions: 2.4.0
Reporter: Michael Beckerle
user reports this diagnostic, which in prior revisions of Daffodil would have
had a line number in it.
{code:java}
[error] Parse Error: Assertion failed: { . eq 9 } failed
Schema context: HiddenLabel Location in test.dfdl.xsd
Data location was preceding byte 1
{code}
this is pretty egregious if no assert failures get line numbers any more.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2183) Unparse nilled complex element fails.
Michael Beckerle created DAFFODIL-2183:
--
Summary: Unparse nilled complex element fails.
Key: DAFFODIL-2183
URL: https://issues.apache.org/jira/browse/DAFFODIL-2183
Project: Daffodil
Issue Type: Bug
Components: Back End
Affects Versions: 2.4.0
Reporter: Michael Beckerle
Fix For: 2.5.0
This schema uses a nillable complex element.
{code:java}
{code}
Parsing this data:
{code:java}
Person:John Doe,29
Person:Sally Smith,34
Person:
Person:Bob Jones,51{code}
That produces this infoset when parsing:
{code:java}
http://www.w3.org/2001/XMLSchema-instance";>
John Doe
29
Sally Smith
34
Bob Jones
51
{code}
Unparsing dies with error:
{code:java}
Unparse Error: Element {}person does not have a value.{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2182) Expression .[1] on array should evaluate to the value of the first element. Produces value of current element.
[
https://issues.apache.org/jira/browse/DAFFODIL-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885336#comment-16885336
]
Michael Beckerle commented on DAFFODIL-2182:
Per review comment from Brandon Sloane:
{quote}{ . } and \{ .[1] } are the same thing when the current node is an
element, aren't they? I don't think "." is supposed to elevate you to the array
containing the current element. I tested this here:
[https://www.freeformatter.com/xpath-tester.html] with the xml:
{{ 1 2 3 }}
where the expression {{foo/a[2]/.[1]}} resolved to 2
{quote}
This issue may have been discussed in DFDL workgroup email archives in the
past. Must search there and/or get clarification about the behavior of .[i].
> Expression .[1] on array should evaluate to the value of the first element.
> Produces value of current element.
> --
>
> Key: DAFFODIL-2182
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2182
> Project: Daffodil
> Issue Type: Bug
> Components: Middle "End"
>Affects Versions: 2.4.0
>Reporter: Michael Beckerle
>Priority: Minor
>
> See test_array_self_expr1.
> I believe when an element is an array, that .[i] should be meaningful.
> It appears that the DPath compiler is creating an implementation that just
> gets the value of the current element ignoring the indexing.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (DAFFODIL-2182) Expression .[1] on array should evaluate to the value of the first element. Produces value of current element.
Michael Beckerle created DAFFODIL-2182: -- Summary: Expression .[1] on array should evaluate to the value of the first element. Produces value of current element. Key: DAFFODIL-2182 URL: https://issues.apache.org/jira/browse/DAFFODIL-2182 Project: Daffodil Issue Type: Bug Components: Middle "End" Affects Versions: 2.4.0 Reporter: Michael Beckerle See test_array_self_expr1. I believe when an element is an array, that .[i] should be meaningful. It appears that the DPath compiler is creating an implementation that just gets the value of the current element ignoring the indexing. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (DAFFODIL-2180) experimental features need properties to enable/disable
[ https://issues.apache.org/jira/browse/DAFFODIL-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle closed DAFFODIL-2180. -- > experimental features need properties to enable/disable > --- > > Key: DAFFODIL-2180 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2180 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > Experimental features are supposed to have a property that enables/disables > them. > Layering does not currently have such a property. > TypeCalc and enums feature also needs to have such. > hexBinary with length in bits also. > There may be others. They should all get added en-masse, not one by one. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (DAFFODIL-2180) experimental features need properties to enable/disable
[ https://issues.apache.org/jira/browse/DAFFODIL-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-2180. Resolution: Duplicate Fix Version/s: (was: 2.5.0) 2.4.0 Duplicate of DAFFODIL-2170 which has been renamed to cover all experimental features. > experimental features need properties to enable/disable > --- > > Key: DAFFODIL-2180 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2180 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > Experimental features are supposed to have a property that enables/disables > them. > Layering does not currently have such a property. > TypeCalc and enums feature also needs to have such. > hexBinary with length in bits also. > There may be others. They should all get added en-masse, not one by one. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DAFFODIL-2170) experimental features (e.g., hexBinary with lengthUnits 'bits') need property to enable/disable.
[ https://issues.apache.org/jira/browse/DAFFODIL-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883216#comment-16883216 ] Michael Beckerle commented on DAFFODIL-2170: All the experimental features need this same enable/disable treatment: layering typeCalc and enums etc. > experimental features (e.g., hexBinary with lengthUnits 'bits') need property > to enable/disable. > > > Key: DAFFODIL-2170 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > > The DFDL Workgroup has concluded that this feature is on the list for DFDL > v2.0. (Few things that don't have multiple implementations are going to be > added to DFDL v1.0 any more.) > That means this has to become an experimental feature which means it has to > have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (DAFFODIL-2170) experimental features (e.g., hexBinary with lengthUnits 'bits') need property to enable/disable.
[ https://issues.apache.org/jira/browse/DAFFODIL-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2170: --- Summary: experimental features (e.g., hexBinary with lengthUnits 'bits') need property to enable/disable. (was: hexBinary with lengthUnits 'bits' needs experimental property to enable/disable.) > experimental features (e.g., hexBinary with lengthUnits 'bits') need property > to enable/disable. > > > Key: DAFFODIL-2170 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > > The DFDL Workgroup has concluded that this feature is on the list for DFDL > v2.0. (Few things that don't have multiple implementations are going to be > added to DFDL v1.0 any more.) > That means this has to become an experimental feature which means it has to > have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2180) experimental features need properties to enable/disable
Michael Beckerle created DAFFODIL-2180: -- Summary: experimental features need properties to enable/disable Key: DAFFODIL-2180 URL: https://issues.apache.org/jira/browse/DAFFODIL-2180 Project: Daffodil Issue Type: Improvement Components: Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.5.0 Experimental features are supposed to have a property that enables/disables them. Layering does not currently have such a property. TypeCalc and enums feature also needs to have such. hexBinary with length in bits also. There may be others. They should all get added en-masse, not one by one. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (DAFFODIL-2179) Add doc page for dfdlx extension features
Michael Beckerle created DAFFODIL-2179: -- Summary: Add doc page for dfdlx extension features Key: DAFFODIL-2179 URL: https://issues.apache.org/jira/browse/DAFFODIL-2179 Project: Daffodil Issue Type: Improvement Components: Documentation Reporter: Michael Beckerle Daffodil has numerous dfdlx features. These do have documentation scattered on the developers wiki under design notes, and intermixed with other things that aren't documentation users would need. Users aren't going to be able to find this. Since these are dfdlx features, a wiki page with links to their doc is the right thing, but we should have a link on the user web site that reaches this summary page, with links to all the dfdlx extension page wiki pages. Copious warnings about "subject to change without backward compatibility support." can then be clearly displayed, and the dfdlx features can be grouped into those we want people to experiement with, and those we don't (which we might possibly just decide not to mention at all.) The above is intended to add minimum work to adding a dfdlx extension to DFDL while giving users a clue about dfdlx features they may see in a DFDL schema, or need to use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DAFFODIL-2175) link to calabash is to nifi page
Michael Beckerle created DAFFODIL-2175: -- Summary: link to calabash is to nifi page Key: DAFFODIL-2175 URL: https://issues.apache.org/jira/browse/DAFFODIL-2175 Project: Daffodil Issue Type: Bug Components: Documentation Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.4.0 The daffodil site has a link to XML Calabash from the Getting Started page which is actually to the NiFi integration. We should fix this, or redirect people to the parent where all the NCSA-hosted extensions are. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-2170) hexBinary with lengthUnits 'bits' needs experimental property to enable/disable.
[ https://issues.apache.org/jira/browse/DAFFODIL-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875069#comment-16875069 ] Michael Beckerle commented on DAFFODIL-2170: Good point. Using a property is problematic because if the property is in the dfdlx namespace, then well we're sharing a namespace. So we'd need to choose a property name that is specific to daffodil like dfdlx:daffodilCompatibilityMode, or dfdlx:daffodilModeFlags="extensionsEnabled" or something like that. It is supposed to be clear looking at a schema if it uses extension features. Some enablement on the top level dfdl:format annotation is supposed to be required. (I'm supposed to write up this whole extensions feature as a DFDL Workgroup document.) This is supposed ot require more than just seeing the dfdlx prefix and namespace defined on the xsd:schema element. Just the presence of that isn't enough. I think a clearly daffodil named property on the dfdl:format annotation should be needed to turn on extensions - eventually that will be required. Initially to avoid breaking things we'll have the usual default from tunable provides setting of this, and warnings - currently suppressed if extensions are found, but the property is being defaulted because it is missing. > hexBinary with lengthUnits 'bits' needs experimental property to > enable/disable. > > > Key: DAFFODIL-2170 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > The DFDL Workgroup has concluded that this feature is on the list for DFDL > v2.0. (Few things that don't have multiple implementations are going to be > added to DFDL v1.0 any more.) > That means this has to become an experimental feature which means it has to > have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-2170) hexBinary with lengthUnits 'bits' needs experimental property to enable/disable.
[ https://issues.apache.org/jira/browse/DAFFODIL-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874494#comment-16874494 ] Michael Beckerle commented on DAFFODIL-2170: I agree. A bunch of dfdlx features could be controlled by a single mode flag property. dfdlx:compatibilityMode is a fine property name. Not sure I like "experimental", as that suggests these features are not even proposed yet. For example, the hexBinary with length bits is very solid. The layering less so (e.g., can't do checksums yet, but we know people want to do those), and the enum/type-calc stuff even less. > hexBinary with lengthUnits 'bits' needs experimental property to > enable/disable. > > > Key: DAFFODIL-2170 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > The DFDL Workgroup has concluded that this feature is on the list for DFDL > v2.0. (Few things that don't have multiple implementations are going to be > added to DFDL v1.0 any more.) > That means this has to become an experimental feature which means it has to > have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DAFFODIL-2170) hexBinary with lengthUnits 'bits' needs experimental property to enable/disable.
[ https://issues.apache.org/jira/browse/DAFFODIL-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874494#comment-16874494 ] Michael Beckerle edited comment on DAFFODIL-2170 at 6/27/19 8:41 PM: - I agree. A bunch of dfdlx features could be controlled by a single mode flag property. dfdlx:compatibilityMode is a fine property name. was (Author: mbeckerle): I agree. A bunch of dfdlx features could be controlled by a single mode flag property. dfdlx:compatibilityMode is a fine property name. Not sure I like "experimental", as that suggests these features are not even proposed yet. For example, the hexBinary with length bits is very solid. The layering less so (e.g., can't do checksums yet, but we know people want to do those), and the enum/type-calc stuff even less. > hexBinary with lengthUnits 'bits' needs experimental property to > enable/disable. > > > Key: DAFFODIL-2170 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 > Project: Daffodil > Issue Type: Improvement > Components: Front End >Affects Versions: 2.4.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > The DFDL Workgroup has concluded that this feature is on the list for DFDL > v2.0. (Few things that don't have multiple implementations are going to be > added to DFDL v1.0 any more.) > That means this has to become an experimental feature which means it has to > have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DAFFODIL-2170) hexBinary with lengthUnits 'bits' needs experimental property to enable/disable.
Michael Beckerle created DAFFODIL-2170: -- Summary: hexBinary with lengthUnits 'bits' needs experimental property to enable/disable. Key: DAFFODIL-2170 URL: https://issues.apache.org/jira/browse/DAFFODIL-2170 Project: Daffodil Issue Type: Improvement Components: Front End Affects Versions: 2.4.0 Reporter: Michael Beckerle Fix For: 2.4.0 The DFDL Workgroup has concluded that this feature is on the list for DFDL v2.0. (Few things that don't have multiple implementations are going to be added to DFDL v1.0 any more.) That means this has to become an experimental feature which means it has to have a property for enabling/disabling it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1493) %ES; delimiter doesn't work right
[ https://issues.apache.org/jira/browse/DAFFODIL-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874320#comment-16874320 ] Michael Beckerle commented on DAFFODIL-1493: Note that DFDL Spec is being changed. %ES; is allowed as the only delimiter but only when lengthKind is NOT 'delimited', so we're not scanning for it (since there is nothing to scan for). > %ES; delimiter doesn't work right > - > > Key: DAFFODIL-1493 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1493 > Project: Daffodil > Issue Type: Bug > Components: Back End, General >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.4.0 > > > See the linked review comments for more details. The jist of it is that when > an %ES; is in a delimiter, the associated DFA is not getting compiled > correctly, and it actually will match a literal %ES;. There is code in > DelimiterParser.scala that inefficiently looks for ES when a match isn't > found, and accepts it as an empty match. Instead, the DFA scanner should be > returning the %ES; as a successful match in the parse results, and then > DelimiterParser.scala can use the same logic to determine which DFA matched. > This would allow us to get rid of the hasES functions and be more efficient. > Note that some things may be tricky because when scanning for a delimiter, we > have to ignore the %ES;, since they'll match anything. We may need new > iterators that ignore these. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1475) Runtime terminator containing isolated %ES; provides poor diagnostic.
[ https://issues.apache.org/jira/browse/DAFFODIL-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874317#comment-16874317 ] Michael Beckerle commented on DAFFODIL-1475: Note that this behavior is to be different. The spec will be corrected to allow %ES; alone as a delimiter, but only when not lengthKind delimited. > Runtime terminator containing isolated %ES; provides poor diagnostic. > - > > Key: DAFFODIL-1475 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1475 > Project: Daffodil > Issue Type: Bug > Components: Back End, Diagnostics, Middle "End", Usability >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Steve Lawrence >Priority: Major > > See test_emptyStringEntityTermInExpression_01 > (which I recently modified to be a negative test, looking for a message about > the incorrect use of %ES; in the terminator.) > If you have an expression for a terminator, and the expression returns a > string containing %ES; by itself, that's not allowed (you can't "turn off" > delimiters at runtime by specifying a delimiter to be empty string) > This should be a runtime SDE. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-2152) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/DAFFODIL-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2152: -- Assignee: (was: Guichard) > Add support for Scala 2.13 > -- > > Key: DAFFODIL-2152 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2152 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > > Scala 2.13 has been released and has some nice updates: > https://github.com/scala/scala/releases/tag/v2.13.0 > We should enable cross compilation support for it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-1535) PCAP: schema use use daf:error() (or perhaps fn:error() based on DFDL-WG feedback)
[ https://issues.apache.org/jira/browse/DAFFODIL-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-1535: -- Assignee: (was: Guichard) > PCAP: schema use use daf:error() (or perhaps fn:error() based on DFDL-WG > feedback) > -- > > Key: DAFFODIL-1535 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1535 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups >Reporter: Steve Lawrence >Priority: Major > > There are some cases in the PCAP schema where outputValueCalc will return -1. > These are really error cases, and daf:error() or fn:error() should really be > used instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1638) Isolate invalid DFDL schema and XML files so IDEs dont flag them
[ https://issues.apache.org/jira/browse/DAFFODIL-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869921#comment-16869921 ] Michael Beckerle commented on DAFFODIL-1638: No so simple. Somehow all of what was described in the comment above has stopped working in Eclipse for me, though it *was* working.. I am still recommending people just leave all XML validation turned off. > Isolate invalid DFDL schema and XML files so IDEs dont flag them > > > Key: DAFFODIL-1638 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1638 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, Infrastructure >Reporter: Michael Beckerle >Priority: Major > > our daffodil source tree has, intentionally, invalid DFDL schema files that > cause validation to fail for unit test purposes (negative tests). We should > rename these files so they don't cause the Eclipse (or other) IDE to create > distracting validation warnings when you have the daffodil source tree open. > These tests need to explicitly use file names like foo.dfdl.xsd.invalid so > that the IDE will not view it as a XSD file and try to validate it. > This will enable a single setup of Eclipse to be used by persons trying to > author DFDL schemas, who want the XML & XSD validation capabilities turned > on, and used by persons trying to do daffodil development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-1879) Compilation related data structures cannot be garbage collected due to RuntimeData strong references
[ https://issues.apache.org/jira/browse/DAFFODIL-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1879: --- Component/s: Front End Back End > Compilation related data structures cannot be garbage collected due to > RuntimeData strong references > > > Key: DAFFODIL-1879 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1879 > Project: Daffodil > Issue Type: Bug > Components: Back End, Front End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.4.0 > > > All the *RuntimeData and DPathCompileInfo constructors accept pass-by-name > parameters. Some of the arguments passed into these constructors reference > objects that are only needed for schema compilation (e.g. dsom, grammar, > etc.). Because they are pass by name, these objects are strong reference and > thus cannot be garbage collected, leading to the entire compilation state > being store in memory. > Fortunately, those parameters are marked as transient and so when serialized > and deserialized they are effectively garbage collected. But if one does not > save/reload a schema, lots of wasted memory used during schema compilation > will stick around. Pass-by-name parameters cannot be vars, so the simple > thing of making them vars and then setting to null does not work. These > objects should be restructured to allow them to be garbage collected once > compiled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-1837) Correct ExternalVariablesLoader to not take DaffodilTunables object in API calls
[ https://issues.apache.org/jira/browse/DAFFODIL-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1837: --- Component/s: API > Correct ExternalVariablesLoader to not take DaffodilTunables object in API > calls > > > Key: DAFFODIL-1837 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1837 > Project: Daffodil > Issue Type: Bug > Components: API >Reporter: Taylor Wise >Priority: Major > Fix For: 2.4.0 > > > We don't want to expose the DaffodilTunables object to the user via the API. > Update ExternalVariablesLoader methods to only accept a Map[String,String] > for tunables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-2124) Error unparsing bigInts
[ https://issues.apache.org/jira/browse/DAFFODIL-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2124: --- Component/s: Unparsing Back End > Error unparsing bigInts > --- > > Key: DAFFODIL-2124 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2124 > Project: Daffodil > Issue Type: Bug > Components: Back End, Unparsing >Reporter: Brandon Sloane >Assignee: Josh Adams >Priority: Major > Attachments: a.bin.xml, a.dfdl.xsd > > > On unparse, daffodil appears to try parsing numeric strings as a long > regardless of the bit length, potentiall causing issues when length>64. > Example attached. > > !! > !! An unexpected exception occurred. This is a bug! !! > !! > Please report this bug and help us fix it: > https://daffodil.apache.org/community/#issue-tracker > Please include the following exception, the command you > ran, and any input, schema, or tdml files used that led > to this bug. > java.lang.NumberFormatException: For input string: "7089904312036126320" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:583) > at java.lang.Integer.parseInt(Integer.java:615) > at scala.collection.immutable.StringLike.toInt(StringLike.scala:301) > at scala.collection.immutable.StringLike.toInt$(StringLike.scala:301) > at scala.collection.immutable.StringOps.toInt(StringOps.scala:29) > at > org.apache.daffodil.dpath.NodeInfo$PrimType$Int$.fromXMLString(NodeInfo.scala:459) > at > org.apache.daffodil.dpath.NodeInfo$PrimType$Int$.fromXMLString(NodeInfo.scala:457) > at > org.apache.daffodil.infoset.InfosetInputter.createElement(InfosetInputter.scala:347) > at > org.apache.daffodil.infoset.InfosetInputter.handleStartElement(InfosetInputter.scala:242) > at > org.apache.daffodil.infoset.InfosetInputter.reallyFill(InfosetInputter.scala:226) > at > org.apache.daffodil.infoset.InfosetInputter.fill(InfosetInputter.scala:213) > at org.apache.daffodil.util.CursorImplMixin.doAdvance(Cursor.scala:174) > at org.apache.daffodil.util.CursorImplMixin.advance(Cursor.scala:144) > at org.apache.daffodil.util.CursorImplMixin.advance$(Cursor.scala:141) > at > org.apache.daffodil.infoset.InfosetInputter.advance(InfosetInputter.scala:53) > at > org.apache.daffodil.processors.unparsers.UStateMain.advance(UState.scala:482) > at org.apache.daffodil.util.Cursor.advanceMaybe(Cursor.scala:104) > at org.apache.daffodil.util.Cursor.advanceMaybe$(Cursor.scala:103) > at > org.apache.daffodil.processors.unparsers.UState.advanceMaybe(UState.scala:72) > at > org.apache.daffodil.processors.unparsers.UStateMain.advanceOrError(UState.scala:499) > at > org.apache.daffodil.processors.unparsers.RegularElementUnparserStartEndStrategy.unparseBegin(ElementUnparser.scala:382) > at > org.apache.daffodil.processors.unparsers.RegularElementUnparserStartEndStrategy.unparseBegin$(ElementUnparser.scala:377) > at > org.apache.daffodil.processors.unparsers.ElementSpecifiedLengthUnparser.unparseBegin(ElementUnparser.scala:264) > at > org.apache.daffodil.processors.unparsers.ElementUnparserBase.unparse(ElementUnparser.scala:178) > at > org.apache.daffodil.processors.unparsers.Unparser.unparse1(Unparser.scala:72) > at > org.apache.daffodil.processors.unparsers.Unparser.unparse1$(Unparser.scala:39) > at > org.apache.daffodil.processors.unparsers.CombinatorUnparser.unparse1(Unparser.scala:128) > at org.apache.daffodil.processors.DataProcessor.doUnparse(Runtime.scala:356) > at org.apache.daffodil.processors.DataProcessor.unparse(Runtime.scala:304) > at org.apache.daffodil.Main$.$anonfun$run$14(Main.scala:1133) > at org.apache.daffodil.util.Timer$.getTimeResult(Timer.scala:76) > at org.apache.daffodil.util.Timer$.getResult(Timer.scala:35) > at org.apache.daffodil.Main$.run(Main.scala:1133) > at org.apache.daffodil.Main$.main(Main.scala:1361) > at org.apache.daffodil.Main.main(Main.scala) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-2163) False warning about unused dfdl:choiceBranchKey on direct dispatch array
[ https://issues.apache.org/jira/browse/DAFFODIL-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2163: --- Component/s: Front End > False warning about unused dfdl:choiceBranchKey on direct dispatch array > > > Key: DAFFODIL-2163 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2163 > Project: Daffodil > Issue Type: Bug > Components: Front End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > > This seems potentially related to DAFFODIL-2162. That issue was resolved, but > it now results in a warning about dfdl:choiceBranchKey property is unused. > The tests clearly show that direct dispatch works so the branch key isn't > actually usnused. Perhaps the property is being cached on something that we > aren't looking at to see which properties have been used, or the property is > being used in a way that isn't caching at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-2153) Stupidly long DPath expression triggers uncaught stack overflow
[
https://issues.apache.org/jira/browse/DAFFODIL-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-2153:
---
Component/s: Front End
Back End
> Stupidly long DPath expression triggers uncaught stack overflow
> ---
>
> Key: DAFFODIL-2153
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2153
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, Front End
>Reporter: Brandon Sloane
>Priority: Major
> Attachments: a.dfdl.xsd
>
>
> The attached schema triggers an uncaught StackOverflowError.
> The attached schema happens to be invalid due to ripping out just the
> offending line (which now references non-existing elements). Ignoring those
> problems, it is probably still reasonable to refuse to compile the attached
> schema just because the expression is so long that I suspect the runtime
> performance to be such that no one actually will want to use it.
>
> Interestingly, attempting to remove the dfdl:outputTypeCalcInt turns this
> into an SDE. This might be because I accidently mis-balanced paranthesies
> allowing the parser to fail normally before overflowing.
> {quote}Exception in thread "main" java.lang.StackOverflowError
> at
> org.apache.daffodil.dpath.FunctionCallExpression.functionObject$lzycompute(Expression.scala:1333)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.functionObject(Expression.scala:1333)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.text(Expression.scala:1314)
> at
> org.apache.daffodil.dpath.ParenthesizedExpression.text(Expression.scala:2203)
> at
> org.apache.daffodil.dpath.ParenthesizedExpression.text(Expression.scala:2203)
> at org.apache.daffodil.dpath.IfExpression.text(Expression.scala:651)
> at
> org.apache.daffodil.dpath.FunctionCallBase.$anonfun$text$2(Expression.scala:1711)
> at scala.collection.immutable.List.map(List.scala:283)
> at org.apache.daffodil.dpath.FunctionCallBase.text(Expression.scala:1711)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.text(Expression.scala:1314)
> at
> org.apache.daffodil.dpath.FunctionCallBase.$anonfun$text$2(Expression.scala:1711)
> at scala.collection.immutable.List.map(List.scala:283)
> at org.apache.daffodil.dpath.FunctionCallBase.text(Expression.scala:1711)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.text(Expression.scala:1314)
> at
> org.apache.daffodil.dpath.FunctionCallBase.$anonfun$text$2(Expression.scala:1711)
> at scala.collection.immutable.List.map(List.scala:283)
> at org.apache.daffodil.dpath.FunctionCallBase.text(Expression.scala:1711)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.text(Expression.scala:1314)
> at org.apache.daffodil.dpath.WholeExpression.text(Expression.scala:576)
> at
> org.apache.daffodil.dpath.Expression.wholeExpressionText$lzycompute(Expression.scala:146)
> at
> org.apache.daffodil.dpath.Expression.wholeExpressionText(Expression.scala:146)
> at org.apache.daffodil.dpath.StepExpression.relPathErr(Expression.scala:849)
> at org.apache.daffodil.dpath.Up.$anonfun$stepElement$8(Expression.scala:1025)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.daffodil.dpath.Up.stepElement$lzycompute(Expression.scala:1025)
> at org.apache.daffodil.dpath.Up.stepElement(Expression.scala:1013)
> at
> org.apache.daffodil.dpath.NamedStep.$anonfun$stepElement$18(Expression.scala:1151)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.daffodil.dpath.NamedStep.stepElement$lzycompute(Expression.scala:1150)
> at org.apache.daffodil.dpath.NamedStep.stepElement(Expression.scala:1134)
> at
> org.apache.daffodil.dpath.StepExpression.inherentType$lzycompute(Expression.scala:946)
> at
> org.apache.daffodil.dpath.StepExpression.inherentType(Expression.scala:943)
> at
> org.apache.daffodil.dpath.PathExpression.inherentType$lzycompute(Expression.scala:721)
> at
> org.apache.daffodil.dpath.PathExpression.inherentType(Expression.scala:721)
> at
> org.apache.daffodil.dpath.ParenthesizedExpression.inherentType(Expression.scala:2201)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.functionObject$lzycompute(Expression.scala:1563)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.functionObject(Expression.scala:1333)
> at
> org.apache.daffodil.dpath.FunctionCallExpression.text(Expression.scala:1314)
> at
> org.apache.daffodil.dpath.ParenthesizedExpression.text(Expression.scala:2203)
> at
> org.apache.daffodil.dpath.ParenthesizedExpression.text(Expression.scala:2203)
> at org.apache.daffodil.dpath.IfExpression.text(Expression.scala:651)
> at
> org.apache.daffodil.dpath.FunctionCallBase.$anonfun$text$2(Expression.scala:1711)
> at scala.collection.immutable.List.map(List.scala:283)
> at org.apache.daffodil.d
[jira] [Updated] (DAFFODIL-2030) Add support for nested prefix lengths
[ https://issues.apache.org/jira/browse/DAFFODIL-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-2030: --- Component/s: Front End Back End > Add support for nested prefix lengths > - > > Key: DAFFODIL-2030 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2030 > Project: Daffodil > Issue Type: New Feature > Components: Back End, Front End >Affects Versions: 2.3.0 >Reporter: Steve Lawrence >Priority: Minor > > DAFFODIL-114 added support for prefixed lengths, but did not implemented > nested prefix lengths. Parsing should work without any modifications (one the > nested SDE is removed) but unparsing still requires updates. The difficulty > is that nested prefix lengths results in nested suspensions, which is not > supported in the current unparse design. All suspensions split off of the > main state. We either need to support nested suspensions, or figure out a way > to make it so the suspensions aren'tnested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-2163) False warning about unused dfdl:choiceBranchKey on direct dispatch array
[ https://issues.apache.org/jira/browse/DAFFODIL-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-2163: -- Assignee: Steve Lawrence > False warning about unused dfdl:choiceBranchKey on direct dispatch array > > > Key: DAFFODIL-2163 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2163 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > > This seems potentially related to DAFFODIL-2162. That issue was resolved, but > it now results in a warning about dfdl:choiceBranchKey property is unused. > The tests clearly show that direct dispatch works so the branch key isn't > actually usnused. Perhaps the property is being cached on something that we > aren't looking at to see which properties have been used, or the property is > being used in a way that isn't caching at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1638) Isolate invalid DFDL schema and XML files so IDEs dont flag them
[ https://issues.apache.org/jira/browse/DAFFODIL-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869805#comment-16869805 ] Michael Beckerle commented on DAFFODIL-1638: So validation of DFDL schemas seems hopeless in eclipse (at least without modifying eclipse) for now. But one can still achieve something and get some good eclipse UI support. One can enable XML validation, setup ".tdml" files to be viewed as XML, and then a number of small bugs arise which can be fixed where we have xml files that are invalid for small reasons. What works: * content assist for long-form dfdl properties. E.g., if you place cursor in a dfdl:format or dfdl:element annotation element, then eclipse provides an "Add-Attribute" menu which will roll-out and show you all the allowed properties. It will even insert them with the "dfdl:" prefix for short form inside embedded schemas of TDML files. * content assist for TDML files. * Validation of TDML files * Validation of XML files Once you do this, the test TDML files that intentionally contain invalid DFDL schema become apparent. There are 28 validation errors spread over 8 TDML files in daffodil-test Once you enable XML validation, TDML files must declare namespace prefixes for ex and tns, because the eclipse tooling can't see the magic the TDML runner does to automatically insert these. Probably the TDML runner feature that does this magic should be deprecated so that we're not creating TDML files that are invalid relative to the tdml.xsd schema. > Isolate invalid DFDL schema and XML files so IDEs dont flag them > > > Key: DAFFODIL-1638 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1638 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, Infrastructure >Reporter: Michael Beckerle >Priority: Major > > our daffodil source tree has, intentionally, invalid DFDL schema files that > cause validation to fail for unit test purposes (negative tests). We should > rename these files so they don't cause the Eclipse (or other) IDE to create > distracting validation warnings when you have the daffodil source tree open. > These tests need to explicitly use file names like foo.dfdl.xsd.invalid so > that the IDE will not view it as a XSD file and try to validate it. > This will enable a single setup of Eclipse to be used by persons trying to > author DFDL schemas, who want the XML & XSD validation capabilities turned > on, and used by persons trying to do daffodil development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DAFFODIL-1638) Isolate invalid DFDL schema and XML files so IDEs dont flag them
[ https://issues.apache.org/jira/browse/DAFFODIL-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869754#comment-16869754 ] Michael Beckerle edited comment on DAFFODIL-1638 at 6/21/19 6:03 PM: - The state of the art in the open-source eclipse & ScalaIDE flavor thereof seems unable to address this problem. If XML Schema Validation is on, then Eclipse cannot use the relative path schemaLocation attributes in xs:import or xs:include to find the corresponding files on the classpath. Nor can the XML catalog simply override them. It simply doesn't resolve them in a compatible manner. Nor can you put a schema location identified by a relative path in the XML catalog - it insists on an absolute path, but those don't appear in any of our DFDL schema files as schemaLocations. It seems there just is no consensus in the XML world about how XML Schema schemaLocation is used/resolved, whether it can contain relative paths or not, etc. If XML Schema Validation is off, and you add "dfdl.xsd" as an extension for "regular" XML validation, eclipse seems to not validate the DFDL schemas as if they were ordinary XML. Even if it did, it would not be treating them as XML Schemas, and so wouldn't process the includes/imports, so would not have everything it needs to validate, and it wouldn't know about QName references and how to validate them. It's not very helpful to validate a DFDL schema as just an XML document well it would help deal with editing and completion, and validation of properties, but in any case, it doesn't work. It seems eclipse special cases the "xsd" extension no matter what. The implications of this are that you simply cannot have XML nor XML Schema validation turned on in the preferences. These must be shut off. Content-assist for TDML files does still work in eclipse if the XML catalog has entries for the DFDL URI and TDML URI. So something works. But validation has to be off to avoid getting dozens/hundreds of spurious errors. was (Author: mbeckerle): The state of the art in the open-source eclipse & ScalaIDE flavor thereof seems unable to address this problem. If XML Schema Validation is on, then Eclipse cannot use the relative path schemaLocation attributes in xs:import or xs:include to find the corresponding files on the classpath. Nor can the XML catalog simply override them. It simply doesn't resolve them in a compatible manner. Nor can you put a schema location identified by a relative path in the XML catalog - it insists on an absolute path, but those don't appear in any of our DFDL schema files as schemaLocations. It seems there just is no consensus in the XML world about how XML Schema schemaLocation is used/resolved, whether it can contain relative paths or not, etc. If XML Schema Validation is off, and you add "dfdl.xsd" as an extension for "regular" XML validation, eclipse seems to not validate the DFDL schemas as if they were ordinary XML. Even if it did, it would not be treating them as XML Schemas, and so wouldn't process the includes/imports, so would not have everything it needs to validate, and it wouldn't know about QName references and how to validate them. It's not very helpful to validate a DFDL schema as just an XML document well it would help deal with editing and completion, and validation of properties, but in any case, it doesn't work. It seems eclipse special cases the "xsd" extension no matter what. > Isolate invalid DFDL schema and XML files so IDEs dont flag them > > > Key: DAFFODIL-1638 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1638 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, Infrastructure >Reporter: Michael Beckerle >Priority: Major > > our daffodil source tree has, intentionally, invalid DFDL schema files that > cause validation to fail for unit test purposes (negative tests). We should > rename these files so they don't cause the Eclipse (or other) IDE to create > distracting validation warnings when you have the daffodil source tree open. > These tests need to explicitly use file names like foo.dfdl.xsd.invalid so > that the IDE will not view it as a XSD file and try to validate it. > This will enable a single setup of Eclipse to be used by persons trying to > author DFDL schemas, who want the XML & XSD validation capabilities turned > on, and used by persons trying to do daffodil development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1638) Isolate invalid DFDL schema and XML files so IDEs dont flag them
[ https://issues.apache.org/jira/browse/DAFFODIL-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869754#comment-16869754 ] Michael Beckerle commented on DAFFODIL-1638: The state of the art in the open-source eclipse & ScalaIDE flavor thereof seems unable to address this problem. If XML Schema Validation is on, then Eclipse cannot use the relative path schemaLocation attributes in xs:import or xs:include to find the corresponding files on the classpath. Nor can the XML catalog simply override them. It simply doesn't resolve them in a compatible manner. Nor can you put a schema location identified by a relative path in the XML catalog - it insists on an absolute path, but those don't appear in any of our DFDL schema files as schemaLocations. It seems there just is no consensus in the XML world about how XML Schema schemaLocation is used/resolved, whether it can contain relative paths or not, etc. If XML Schema Validation is off, and you add "dfdl.xsd" as an extension for "regular" XML validation, eclipse seems to not validate the DFDL schemas as if they were ordinary XML. Even if it did, it would not be treating them as XML Schemas, and so wouldn't process the includes/imports, so would not have everything it needs to validate, and it wouldn't know about QName references and how to validate them. It's not very helpful to validate a DFDL schema as just an XML document well it would help deal with editing and completion, and validation of properties, but in any case, it doesn't work. It seems eclipse special cases the "xsd" extension no matter what. > Isolate invalid DFDL schema and XML files so IDEs dont flag them > > > Key: DAFFODIL-1638 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1638 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, Infrastructure >Reporter: Michael Beckerle >Priority: Major > > our daffodil source tree has, intentionally, invalid DFDL schema files that > cause validation to fail for unit test purposes (negative tests). We should > rename these files so they don't cause the Eclipse (or other) IDE to create > distracting validation warnings when you have the daffodil source tree open. > These tests need to explicitly use file names like foo.dfdl.xsd.invalid so > that the IDE will not view it as a XSD file and try to validate it. > This will enable a single setup of Eclipse to be used by persons trying to > author DFDL schemas, who want the XML & XSD validation capabilities turned > on, and used by persons trying to do daffodil development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DAFFODIL-897) TDML runner must check tdml:document bitOrder usage in test cases
[ https://issues.apache.org/jira/browse/DAFFODIL-897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-897. --- Resolution: Fixed Fix Version/s: 2.0.0 This ticket slipped through the cracks. It was implemented years ago and has been a feature of the TDML runner since work on the mil-std-2045 DFDL schema was done back in 2013. Example of this is test_MIL2045_47001D_1 > TDML runner must check tdml:document bitOrder usage in test cases > - > > Key: DAFFODIL-897 > URL: https://issues.apache.org/jira/browse/DAFFODIL-897 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure, QA, TDML Runner >Affects Versions: s11 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.0.0 > > > See DFDL-896 for bitOrder feature. > In order to test this feature, the ability to specify data in binary, but > with least-significant bit first is needed. Otherwise we'll be unable to > create sufficient test cases. > There are two possible ways this can work. > Consider a single byte with 3 bit fields within it, a 3-bit-long field > containing value 3, or binary 011, another 3-bit-long field containing value > 4 or binary 100, and a 2-bit-long field containing value 1 or binary 01. > The TDML of Daffodil today only supports most-significant-bit-first bit order > so: > 011 100 01 > That is equivalent to the byte 0x71 hex. > For least-significant-bit-first, there are a few possibilities. > One possibility is to add a bitOrder="MSBFirst/LSBFirst" attribute to > documentPart. The default would be MSBFirst, which is consistent with today's > behavior. The new value of LSBFirst would mean that we process the bits line > by line, and concatenate them in LSB First order, though each line is written > MSB-first order (the normal way we write numbers, i.e., where 011 binary > means 3. This is consistent with some dump tools we have seen. So the same 3 > fields of data would be: > > x011 first field > xx100xxx second field > 01xx third field. > > Notice that the 'x' characters are ignored. So this is equivalent to writing > > 01 100 011 first second and third > > But see how the bits on the line are interpreted as numbered from right to > left. Hence, viewed as a numeric byte in the data, this would be hex C6. > The bitOrder attribute should be usable with any documentPart type, not just > bits. So equivalent to the above: > > C6 first second and third > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DAFFODIL-1431) DFDL4S compatibility testing
[ https://issues.apache.org/jira/browse/DAFFODIL-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-1431. Resolution: Won't Fix Fix Version/s: 2.4.0 Removing from tickets. This is a wish list item, and should be discussed on the developers wiki or dev@ email list, not be a ticket. > DFDL4S compatibility testing > > > Key: DAFFODIL-1431 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1431 > Project: Daffodil > Issue Type: Wish > Components: QA >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > This task is to take the DFDL4S schema, or suitable part of it, get some > sample data from ESA (European Space Agency), and try it on Daffodil. > As of this writing, the DFDL4S schema language has some incompatibilities > with official DFDL, so the schema will have to be massaged to be standard, > but this should be straightforward to do, if perhaps a bit tedious, and once > done, it should work. > The modified schema could then be incorporated into our user-submitted tests > area. (Depends on whether ESA minds if we do this. They do keep some control > on this software, and that may include the schema. But if we have a partial > schema maybe ok?) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1431) DFDL4S compatibility testing
[ https://issues.apache.org/jira/browse/DAFFODIL-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle closed DAFFODIL-1431. -- > DFDL4S compatibility testing > > > Key: DAFFODIL-1431 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1431 > Project: Daffodil > Issue Type: Wish > Components: QA >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.4.0 > > > This task is to take the DFDL4S schema, or suitable part of it, get some > sample data from ESA (European Space Agency), and try it on Daffodil. > As of this writing, the DFDL4S schema language has some incompatibilities > with official DFDL, so the schema will have to be massaged to be standard, > but this should be straightforward to do, if perhaps a bit tedious, and once > done, it should work. > The modified schema could then be incorporated into our user-submitted tests > area. (Depends on whether ESA minds if we do this. They do keep some control > on this software, and that may include the schema. But if we have a partial > schema maybe ok?) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-1300) Fix Test Scala files to not create the test suite repeatedly
[
https://issues.apache.org/jira/browse/DAFFODIL-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Beckerle updated DAFFODIL-1300:
---
Labels: beginner (was: )
> Fix Test Scala files to not create the test suite repeatedly
>
>
> Key: DAFFODIL-1300
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1300
> Project: Daffodil
> Issue Type: Improvement
> Components: QA
>Reporter: Michael Beckerle
>Priority: Major
> Labels: beginner
>
> Running tests seemed to be awfully slow.
> So I learned today that for every single test case line in our test scala
> files (e.g, TestFacets.scala), it reloads the entire test suite TDML from
> scratch. This makes the tests much slower of course, particularly so for the
> large TDML files (Functions.tdml for example).
> The fix to this is to look at how TestFacets.scala is now (just pushed to
> 1.1.0 branch).
> This is the recipe for loading up the test suite exactly once, but also
> releasing it at the end of the job so that these things don't build up
> endlessly in memory. Basically, all those variables that hold the test runner
> objects, all those go in the associated peer object, not in the test class.
> E.g., from TestFacets.scala:
> object TestFacets {
> val testDir = "/edu/illinois/ncsa/daffodil/section05/facets/"
> val aa = testDir + "Facets.tdml"
> var runner = new DFDLTestSuite(Misc.getRequiredResource(aa),
> validateTDMLFile =
> false, validateDFDLSchemas = false)
> /**
>* Avoid memory leak of adding more and more test suites to static objects
> as we run more and more test suites.
>*/
> @AfterClass def tearDown() { runner = null }
> }
> class TestFacets {
> import TestFacets._ // Imports all of the definitions in the object so we
> don't have to edit the test lines.
> @Test def test_minMaxInExdateTime01() {
> runner.runOneTest("minMaxInExdateTime01") }
> ...
> }
> Tests run much faster once this is done.
> For now I have retained noisy messages which illustrate how often we are
> reloading the TDML files to construct the test suite objects.
> There are also warnings to stderr for duplicate test cases in the same tdml
> file -(see JIRA Ticket DFDL-1298 and DFDL-1299. )
> We should make a pass and fix all the TDML running scala test files to use
> this technique.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (DAFFODIL-902) Fix tests that fail due to dfdl schema validation.
[ https://issues.apache.org/jira/browse/DAFFODIL-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle resolved DAFFODIL-902. --- Resolution: Duplicate Fix Version/s: 2.4.0 Duplicate of DAFFODIL-1638 > Fix tests that fail due to dfdl schema validation. > -- > > Key: DAFFODIL-902 > URL: https://issues.apache.org/jira/browse/DAFFODIL-902 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure, QA >Reporter: Taylor Wise >Priority: Major > Fix For: 2.4.0 > > > Perhaps the right thing is to use a flag like this for now, but just add a > jira task to find and fix all of the tests that have this. > Locate tests that have the validateDFDLSchemas flag set to false. Correct > the tests that fail when setting the flag to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-1638) Isolate invalid DFDL schema and XML files so IDEs dont flag them
[ https://issues.apache.org/jira/browse/DAFFODIL-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-1638: --- Summary: Isolate invalid DFDL schema and XML files so IDEs dont flag them (was: rename invalid DFDL schema and XML files so IDEs dont flag them) > Isolate invalid DFDL schema and XML files so IDEs dont flag them > > > Key: DAFFODIL-1638 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1638 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, Infrastructure >Reporter: Michael Beckerle >Priority: Major > > our daffodil source tree has, intentionally, invalid DFDL schema files that > cause validation to fail for unit test purposes (negative tests). We should > rename these files so they don't cause the Eclipse (or other) IDE to create > distracting validation warnings when you have the daffodil source tree open. > These tests need to explicitly use file names like foo.dfdl.xsd.invalid so > that the IDE will not view it as a XSD file and try to validate it. > This will enable a single setup of Eclipse to be used by persons trying to > author DFDL schemas, who want the XML & XSD validation capabilities turned > on, and used by persons trying to do daffodil development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-966) Add negative tests to performance suite
[ https://issues.apache.org/jira/browse/DAFFODIL-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle updated DAFFODIL-966: -- Priority: Minor (was: Major) > Add negative tests to performance suite > --- > > Key: DAFFODIL-966 > URL: https://issues.apache.org/jira/browse/DAFFODIL-966 > Project: Daffodil > Issue Type: Wish > Components: QA >Reporter: Jessie Chab >Priority: Minor > > Add tests to performance repo for each data type to demonstrate how long the > parser takes to abort or recover from invalid data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-733) CLI Automation: Automate all existing CLI tests for both Windows and Linux
[ https://issues.apache.org/jira/browse/DAFFODIL-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-733: - Assignee: Mark C. Otto (was: Michael Beckerle) > CLI Automation: Automate all existing CLI tests for both Windows and Linux > -- > > Key: DAFFODIL-733 > URL: https://issues.apache.org/jira/browse/DAFFODIL-733 > Project: Daffodil > Issue Type: Wish > Components: QA, Windows >Reporter: Jessie Chab >Assignee: Mark C. Otto >Priority: Major > > We have the initial infrastructure and some CLI Automation tests in the repo. > We need to make sure all existing manual tests are converted to automated > tests (except for those that require more attention than what can be provided > by automation), and that all of these tests work on both Linux and Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-733) CLI Automation: Automate all existing CLI tests for both Windows and Linux
[ https://issues.apache.org/jira/browse/DAFFODIL-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869530#comment-16869530 ] Michael Beckerle commented on DAFFODIL-733: --- Mark, I think this old ticket was done by you a while back. Can you comment on the status of this? > CLI Automation: Automate all existing CLI tests for both Windows and Linux > -- > > Key: DAFFODIL-733 > URL: https://issues.apache.org/jira/browse/DAFFODIL-733 > Project: Daffodil > Issue Type: Wish > Components: QA, Windows >Reporter: Jessie Chab >Assignee: Mark C. Otto >Priority: Major > > We have the initial infrastructure and some CLI Automation tests in the repo. > We need to make sure all existing manual tests are converted to automated > tests (except for those that require more attention than what can be provided > by automation), and that all of these tests work on both Linux and Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-733) CLI Automation: Automate all existing CLI tests for both Windows and Linux
[ https://issues.apache.org/jira/browse/DAFFODIL-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Beckerle reassigned DAFFODIL-733: - Assignee: Michael Beckerle > CLI Automation: Automate all existing CLI tests for both Windows and Linux > -- > > Key: DAFFODIL-733 > URL: https://issues.apache.org/jira/browse/DAFFODIL-733 > Project: Daffodil > Issue Type: Wish > Components: QA, Windows >Reporter: Jessie Chab >Assignee: Michael Beckerle >Priority: Major > > We have the initial infrastructure and some CLI Automation tests in the repo. > We need to make sure all existing manual tests are converted to automated > tests (except for those that require more attention than what can be provided > by automation), and that all of these tests work on both Linux and Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
