stevedlawrence commented on a change in pull request #532:
URL: https://github.com/apache/daffodil/pull/532#discussion_r615970679
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
Review comment:
Is libxml-dev an actual dependeny? I thought we only use mini-xml?
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
+
+ - name: Install Dependencies (Windows)
+ if: runner.os == 'Windows'
+ uses: msys2/setup-msys2@v2
+ with:
+ install: gcc libargp-devel make pkgconf
+ path-type: inherit
- - name: Install Java
- uses: actions/setup-java@v1
+ - name: Check out mxml source (Windows)
+ if: runner.os == 'Windows'
+ uses: actions/[email protected]
+ with:
+ repository: michaelrsweet/mxml
+ ref: v3.2
+ path: mxml
+
+ - name: Install mxml library (Windows)
+ if: runner.os == 'Windows'
+ run: |
+ cd mxml
+ ./configure --prefix=/usr --disable-shared --disable-threads
+ make
+ make install
+ # Workaround for sbt hanging problem
+ echo "COURSIER_CACHE=$temp" >> $GITHUB_ENV
+ echo "COURSIER_CONFIG_DIR=$temp" >> $GITHUB_ENV
Review comment:
Should we be using https://github.com/coursier/cache-action instead? I
assume these lines are doing the same thing? And it seems like a good idea to
cache some build files--might speed up the build a bit if it avoids having to
redownload all our dependencies for every build.
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
+
+ - name: Install Dependencies (Windows)
+ if: runner.os == 'Windows'
+ uses: msys2/setup-msys2@v2
+ with:
+ install: gcc libargp-devel make pkgconf
Review comment:
Looking at libargp on github, I see GPL liceneds files in the repo,
which is category X, which is only allowed if the dependency is optinoal. Is
that the case?
##########
File path: BUILD.md
##########
@@ -0,0 +1,114 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Build Requirements
+
+Daffodil's build requirements include:
+
+* JDK 8 or higher
+* SBT 0.13.8 or higher
+* C compiler C99 or higher
+* Mini-XML Version 3.2 or higher
+
+You will need the Java Software Development Kit ([JDK]) and the Scala
+Build Tool ([SBT]) to build Daffodil, run all tests, create packages,
+and more. You can install the latest [Java 11 LTS][JDK] version and
+the latest [SBT] version following their websites' instructions or
+install them using your operating system's package manager.
+
+Since Daffodil now has a C backend as well as a Scala backend, you
+will need a C compiler supporting the [C99] standard or later, the
+[Mini-XML] library, and possibly the GNU [argp] library if your
+system's C library doesn't include it already. You can install either
+[gcc] or [clang] using your operating system's package manager. If
+you can't install the [Mini-XML] library using your operating system's
+package manager, you'll have to build it from source. We'll tell you
+how to do that on Windows but the commands should work on other
+operating systems too.
+
+You can set your environment variables "CC" and "AR" to the correct
+commands (or set them to `true` to disable C compilation altogether)
+if you don't want `sbt compile` to call your C compiler with `cc` and
+`ar` as the default commands.
+
+## Fedora 33
+
+You can use the `dnf` package manager to install most of the build
+requirements needed to build Daffodil:
+
+ sudo dnf install gcc git java-11-openjdk-devel make mxml-devel pkgconf
+
+However, Fedora has no sbt package in its default repositories.
+You'll have to install the latest [SBT] version following its
+website's instructions.
+
+Now you can build Daffodil from source and the sbt and daffodil
+commands you type will be able to call the C compiler.
+
+## Ubuntu 20.04
+
+You can use the `apt` package manager to install most of the build
+requirements needed to build Daffodil:
+
+ sudo apt install build-essential default-jdk git libmxml-dev
+
+However, Ubuntu has no sbt package in its default repositories.
+You'll have to install the latest [SBT] version following its
+website's instructions.
+
+Now you can build Daffodil from source and the sbt and daffodil
+commands you type will be able to call the C compiler.
+
+## Windows 10
+
+Install the latest [Java 11 LTS][JDK] version and the latest [SBT]
+version following their websites' instructions.
+
+Install [MSYS2] following its website's instructions and open a new
+"MSYS2 MSYS" window. We'll need its collection of free programs and
+libraries.
+
+Install [gcc] and [libargp][argp] using MSYS2's `pacman` package
+manager:
+
+ pacman -S gcc git libargp-devel make pkgconf
+
+However, MSYS2 has no [libmxml-devel][Mini-XML] package so you'll have
Review comment:
libxml is not the same things as Mini-XML, right?
##########
File path: daffodil-cli/src/main/scala/org/apache/daffodil/Main.scala
##########
@@ -502,11 +502,47 @@ class CLIConf(arguments: Array[String]) extends
scallop.ScallopConf(arguments)
val info = tally(descr = "increment test result information output level,
one level for each -i")
}
+ // Generate Subcommand Options
+ object generate extends scallop.Subcommand("generate") {
+ descr("generate <language> code from a DFDL schema")
+
+ banner("""|Usage: daffodil [GLOBAL_OPTS] generate <language>
[SUBCOMMAND_OPTS]
+ |""".stripMargin)
+ shortSubcommandsHelp()
+ footer("""|
+ |Run 'daffodil generate <language> --help' for subcommand
specific options""".stripMargin)
+
+ object c extends scallop.Subcommand("c") {
Review comment:
I'm wondering if there's a benefit it renaming this to something like
"c-fpga"? I believe the generated code is designed to be run on an fpga, so
there are some sacrificies like thread safety that some might expect?
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
##########
@@ -108,6 +109,27 @@ final class ProcessorFactory private(
override def onPath(xpath: String) = sset.onPath(xpath)
+ override def forLanguage(language: String): DFDL.CodeGenerator = {
+ Assert.usage(!isError)
+
+ // Do a poor man's pluggable code generator implementation - we can replace
+ // it after we observe how the validator SPI evolves and wait for our
+ // requirements to become clearer
+ val className = language match {
+ case "c" => "org.apache.daffodil.runtime2.CodeGenerator"
+ case _ => s"code generator; source language $language is not supported"
Review comment:
Can we throw here instead of creating an error message and expecting
that Class.forName to fail?
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
##########
@@ -108,6 +109,27 @@ final class ProcessorFactory private(
override def onPath(xpath: String) = sset.onPath(xpath)
+ override def forLanguage(language: String): DFDL.CodeGenerator = {
+ Assert.usage(!isError)
+
+ // Do a poor man's pluggable code generator implementation - we can replace
+ // it after we observe how the validator SPI evolves and wait for our
+ // requirements to become clearer
+ val className = language match {
+ case "c" => "org.apache.daffodil.runtime2.CodeGenerator"
+ case _ => s"code generator; source language $language is not supported"
+ }
+ import scala.language.existentials // Needed to make next line compile
+ val clazz = Try(Class.forName(className))
+ val constructor = clazz.map { _.getDeclaredConstructor(sset.root.getClass)
}
+ val tryInstance = constructor.map {
_.newInstance(sset.root).asInstanceOf[DFDL.CodeGenerator] }
+ val codeGenerator = tryInstance.recover {
+ case ex => throw new InvalidParserException("Error creating " +
className, ex)
+ }.get
+
Review comment:
Is there an advantage to doing reflection instead of just directly
creating an instance of the runtime2.CodeGenerator? I guess it's the diference
between a nice error message if the daffodil-runtime2 jar isn't on the
classpath? But it's a hard dependency right now, so I wouldn't expect people to
remove that without understanding the consequences. Seems more work is needed
to figure out how to make everything pluggable, so maybe it's not worth this
bit of complication?
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ElementBase.scala
##########
@@ -354,6 +354,38 @@ trait ElementBase
} else DataValue.NoValue
}
+ /**
+ * Is either DataValue[AnyRef] or DataValue.NoValue.
+ *
+ * The value will always be of the matching primitive types for the element,
and
+ * directly usable as the value of a simple-type element.
+ *
+ * When a value is used, it is created from the XSD fixed attribute of the
+ * element declaration, and that string cannot contain DFDL entities of any
kind,
+ * nor any PUA-remapped characters. This insures the fixed value can still be
+ * used for ordinary XML-schema validation outside of Daffodil/DFDL.
+ */
+ final lazy val fixedValue: DataValuePrimitiveNullable = {
+ if (hasFixedValue && (isScalar ||
isArrayWithAtLeastOneRequiredArrayElement)) {
+ val dv = {
Review comment:
Suggest changing to fv, I think dv is just copy pasted from defaultValue?
##########
File path: daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd
##########
@@ -101,6 +101,7 @@
<simpleType name="implementationItem">
<restriction base="xs:token">
<enumeration value="daffodil"/>
+ <enumeration value="daffodil-runtime2"/>
<enumeration value="ibm"/>
Review comment:
Should we even be trying to enumerate implementations? Implementations
are somewhat pluggable, there might be implementations that people might want
to test against that we don't know about?
##########
File path:
daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
##########
@@ -504,6 +504,14 @@
</xs:documentation>
</xs:annotation>
</xs:element>
+ <xs:element name="tdmlImplementation" type="xs:string"
default="daffodil" minOccurs="0">
+ <xs:annotation>
+ <xs:documentation>
+ TDMLDFDLProcessorFactory implementation to use when running TDML
tests.
+ Allowed values are "daffodil" (default), "daffodil-runtime2",
and "ibm".
Review comment:
Same question here, do we want to a define a list of allowed values, or
just assume that users know what they're doing?
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/api/DFDLParserUnparser.scala
##########
@@ -147,11 +149,41 @@ object DFDL {
@deprecated("Use arguments to Compiler.compileSource or compileFile.",
"2.6.0")
def setDistinguishedRootNode(name: String, namespace: String = null): Unit
+ /**
+ * Returns a [[DataProcessor]] to process data matching a compiled XPath
expression
+ * @param xpath XPath expression in DFDL schema that data should match
(you can use only "/" at this time)
+ */
def onPath(xpath: String): DataProcessor
+
+ /**
+ * Returns a [[CodeGenerator]] to generate code from a DFDL schema to
parse or unparse data
+ * @param language source language for generated code (you can use only
"c" at this time)
+ */
+ def forLanguage(language: String): CodeGenerator
}
- trait DataProcessor extends WithDiagnostics {
+ /**
+ * Source code generation and compilation is performed with a
language-specific [[CodeGenerator]],
+ * which must be interrogated for diagnostics to see if each call was
successful or not.
+ */
+ trait CodeGenerator extends WithDiagnostics {
+ /**
+ * Generates language-specific code from a DFDL schema to parse or unparse
data
+ * @param rootNS one of the top-level elements of the DFDL schema (if not
supplied, uses first top-level element)
+ * @param outputDir output directory in which to generate code
+ * @return path of output directory containing generated code
+ */
+ def generateCode(rootNS: Option[RefQName], outputDir: String): os.Path
+
+ /**
+ * Compiles the generated code so it can be used by a TDML test or
something else
+ * @param outputDir path of output directory containing generated code
+ * @return path of executable built from generated code
+ */
+ def compileCode(outputDir: os.Path): os.Path
Review comment:
The name outputDir made me think this was the output directory where the
executable would be built. But it's actually the return of the generatedCode
function? Does it make sense to make it inputDir is the directory to complie,
and also provide a path to the executable so the user has control over where
the executable is built?
##########
File path: daffodil-runtime2/src/main/resources/.clang-format
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+AlignConsecutiveDeclarations: true
+AllowShortFunctionsOnASingleLine: None
+AllowShortIfStatementsOnASingleLine: true
+AlwaysBreakAfterReturnType: TopLevelDefinitions
+BasedOnStyle: llvm
+BreakBeforeBraces: Allman
+ColumnLimit: 110
+IndentWidth: 4
+KeepEmptyLinesAtTheStartOfBlocks: false
Review comment:
Does something like this belong in the root of the repo?
##########
File path: daffodil-runtime2/src/main/resources/c/Makefile
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Here's how to compile the C sources into a program for running
+# parse/unparse tests (.dat <-> .xml)
+
+PROGRAM = ./daffodil
+HEADERS = libcli/*.h libruntime/*.h
+SOURCES = libcli/*.c libruntime/*.c
+INCLUDES = -I libcli -I libruntime
+CFLAGS = -g -Wall -Wextra
+LIBS = -lmxml
+
+$(PROGRAM): $(HEADERS) $(SOURCES)
+ $(CC) $(CFLAGS) $(INCLUDES) $(SOURCES) $(LIBS) -o $(PROGRAM)
+
+# Here's how to run parse/unparse tests (.dat <-> .xml, although you
+# will need to create the .dat and .xml files first)
+
+PARSE_DAT = parse.dat
+UNPARSE_XML = unparse.xml
+
+clean:
+ rm -f $(PROGRAM) test_$(PARSE_DAT) test_$(UNPARSE_XML)
+
+tests: parse-test unparse-test
+
+parse-test: $(PROGRAM)
+ $(PROGRAM) parse $(PARSE_DAT) -o test_$(UNPARSE_XML)
+ xmldiff $(UNPARSE_XML) test_$(UNPARSE_XML)
+
+unparse-test: $(PROGRAM)
+ $(PROGRAM) unparse $(UNPARSE_XML) -o test_$(PARSE_DAT)
+ diff $(PARSE_DAT) test_$(PARSE_DAT)
Review comment:
Is this file included in the output directory when generating code?
Where do parse.dat and unparse.xml come from?
##########
File path: daffodil-runtime2/src/main/resources/examples/NestedUnion.c
##########
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
Review comment:
Should these examples be in src/test, or src/examples? Are these
intended to be distribtued in jars? How are these used? Are they manually
updated everytime we make a change to the generator? Are there tests that
validate that these match generated code?
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/xml_writer.c
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_writer.h"
+#include <assert.h> // for assert
+#include <mxml.h> // for mxmlNewOpaquef, mxml_node_t, mxmlElementSetAttr,
mxmlGetOpaque, mxmlNewElement, mxmlDelete, mxmlGetElement, mxmlNewXML,
mxmlSaveFile, MXML_NO_CALLBACK
+#include <stdbool.h> // for bool
+#include <stdint.h> // for int16_t, int32_t, int64_t, int8_t, uint16_t,
uint32_t, uint64_t, uint8_t
+#include <stdio.h> // for NULL
+#include <string.h> // for strcmp
+#include "errors.h" // for Error, ERR_XML_DECL, ERR_XML_ELEMENT,
ERR_XML_WRITE, LIMIT_XML_NESTING, Error::(anonymous)
+#include "stack.h" // for stack_is_empty, stack_pop, stack_push, stack_top,
stack_init
+
+// Push new XML document on stack (note the stack is stored in a
+// static array which could overflow and stop the program; it also
+// means none of those functions are thread-safe)
+
+static const Error *
+xmlStartDocument(XMLWriter *writer)
+{
+ static mxml_node_t *array[LIMIT_XML_NESTING];
+ stack_init(&writer->stack, array, LIMIT_XML_NESTING);
+
+ mxml_node_t *xml = mxmlNewXML("1.0");
+ if (xml)
+ {
+ stack_push(&writer->stack, xml);
+ return NULL;
+ }
+ else
+ {
+ static Error error = {ERR_XML_DECL, {NULL}};
+ return &error;
+ }
+}
+
+// Pop completed XML document off stack and write it to stream (note
+// stack underflow will stop program)
+
+static const Error *
+xmlEndDocument(XMLWriter *writer)
+{
+ mxml_node_t *xml = stack_pop(&writer->stack);
+ assert(stack_is_empty(&writer->stack));
+
+ int status = mxmlSaveFile(xml, writer->stream, MXML_NO_CALLBACK);
+ if (status < 0)
+ {
+ static Error error = {ERR_XML_WRITE, {NULL}};
+ return &error;
+ }
+ mxmlDelete(xml);
+ return NULL;
+}
+
+// Push new complex element on stack (note stack overflow will stop
+// program)
+
+static const Error *
+xmlStartComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *parent = stack_top(&writer->stack);
+ const char * name = get_erd_name(base->erd);
+ const char * xmlns = get_erd_xmlns(base->erd);
+ mxml_node_t *complex = mxmlNewElement(parent, name);
+ if (xmlns)
+ {
+ const char *ns = get_erd_ns(base->erd);
+ mxmlElementSetAttr(complex, xmlns, ns);
+ }
+ stack_push(&writer->stack, complex);
+ return NULL;
+}
+
+// Pop completed complex element off stack (note stack underflow will
+// stop program)
+
+static const Error *
+xmlEndComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *complex = stack_pop(&writer->stack);
+
+ const char *name_from_xml = mxmlGetElement(complex);
+ const char *name_from_erd = get_erd_name(base->erd);
+ assert(strcmp(name_from_xml, name_from_erd) == 0);
+
+ return NULL;
+}
+
+// Fix a real number to conform to xsd:float syntax if needed
+
+static void
+fixNumberIfNeeded(const char *text)
+{
+ if (text[0] == 'N' && text[1] == 'A')
Review comment:
Shoudl this check that text[2] == 'N' and length is 3? Or does somethign
create "NA" for not a number?
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/api/DFDLParserUnparser.scala
##########
@@ -147,11 +149,41 @@ object DFDL {
@deprecated("Use arguments to Compiler.compileSource or compileFile.",
"2.6.0")
def setDistinguishedRootNode(name: String, namespace: String = null): Unit
+ /**
+ * Returns a [[DataProcessor]] to process data matching a compiled XPath
expression
+ * @param xpath XPath expression in DFDL schema that data should match
(you can use only "/" at this time)
+ */
def onPath(xpath: String): DataProcessor
+
+ /**
+ * Returns a [[CodeGenerator]] to generate code from a DFDL schema to
parse or unparse data
+ * @param language source language for generated code (you can use only
"c" at this time)
+ */
+ def forLanguage(language: String): CodeGenerator
}
- trait DataProcessor extends WithDiagnostics {
+ /**
+ * Source code generation and compilation is performed with a
language-specific [[CodeGenerator]],
+ * which must be interrogated for diagnostics to see if each call was
successful or not.
+ */
+ trait CodeGenerator extends WithDiagnostics {
+ /**
+ * Generates language-specific code from a DFDL schema to parse or unparse
data
+ * @param rootNS one of the top-level elements of the DFDL schema (if not
supplied, uses first top-level element)
+ * @param outputDir output directory in which to generate code
+ * @return path of output directory containing generated code
+ */
+ def generateCode(rootNS: Option[RefQName], outputDir: String): os.Path
Review comment:
Is the returned value always the same as outputDir? Should this just
return a Unit, and one assumes that if no exceptions were thrown then the code
was generated successfully in the provided outputDir?
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
+ // Used by WithDiagnostics methods
+ private var diagnostics: Seq[Diagnostic] = Nil
+ private var errorStatus: Boolean = false
+
+ /**
+ * Writes C source files into a "c" subdirectory of the given output
directory.
+ * Removes the "c" subdirectory if it existed before. Returns the "c"
subdirectory.
+ */
+ override def generateCode(rootNS: Option[RefQName], outputDirArg: String):
os.Path = {
+ // Get the paths of the output directory and its code subdirectory
+ val outputDir = os.Path(Paths.get(outputDirArg).toAbsolutePath)
+ val codeDir = outputDir/"c"
+
+ // Ensure our output directory exists while our code subdirectory does not
+ os.makeDir.all(outputDir)
+ os.remove.all(codeDir)
+
+ // Copy our resource directory and all its C source files to our code
subdirectory
+ // (using synchronized to avoid calling FileSystems.newFileSystem
concurrently)
+ val resourceUri = Misc.getRequiredResource("/c")
Review comment:
Can we namespace this to someting like "/org/apache/daffodi/runtime2/c/"
just to ensure we're gettting the right resource and that some other jar on the
classpath isn't accidentally colliding?
##########
File path: daffodil-runtime2/src/main/resources/c/libruntime/errors.c
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "errors.h"
+#include <assert.h> // for assert
+#include <error.h> // for error
+#include <inttypes.h> // for PRId64
+#include <stdbool.h> // for bool, false, true
+#include <stdio.h> // for NULL, feof, ferror, FILE, size_t
+#include <stdlib.h> // for EXIT_FAILURE
+
+// error_message - return an internationalized error message
+
+static const char *
+error_message(enum ErrorCode code)
+{
+ switch (code)
+ {
+ case ERR_CHOICE_KEY:
+ return "no match between choice dispatch key %" PRId64 " and any
branch key";
+ case ERR_FILE_CLOSE:
+ return "error closing file";
+ case ERR_FILE_FLUSH:
+ return "error flushing stream to file";
+ case ERR_FILE_OPEN:
+ return "error opening file '%s'";
+ case ERR_FIXED_VALUE:
+ return "value of element '%s' does not match value of its "
+ "'fixed' attribute";
+ case ERR_INFOSET_READ:
+ return "cannot read infoset type '%s'";
+ case ERR_INFOSET_WRITE:
+ return "cannot write infoset type '%s'";
+ case ERR_PARSE_BOOL:
+ return "error parsing binary value %" PRId64 " as either true or
false";
+ case ERR_STACK_EMPTY:
+ return "stack empty, stopping program";
+ case ERR_STACK_OVERFLOW:
+ return "stack overflow, stopping program";
+ case ERR_STACK_UNDERFLOW:
+ return "stack underflow, stopping program";
+ case ERR_STREAM_EOF:
+ return "EOF in stream, stopping program";
+ case ERR_STREAM_ERROR:
+ return "error in stream, stopping program";
+ case ERR_STRTOBOOL:
+ return "error converting XML data '%s' to boolean";
+ case ERR_STRTOD_ERRNO:
+ return "error converting XML data '%s' to number";
+ case ERR_STRTOI_ERRNO:
+ return "error converting XML data '%s' to integer";
+ case ERR_STRTONUM_EMPTY:
+ return "found no number in XML data '%s'";
+ case ERR_STRTONUM_NOT:
+ return "found non-number characters in XML data '%s'";
+ case ERR_STRTONUM_RANGE:
+ return "number in XML data '%s' out of range";
+ case ERR_XML_DECL:
+ return "error making new XML declaration";
+ case ERR_XML_ELEMENT:
+ return "error making new XML element '%s'";
+ case ERR_XML_ERD:
+ return "unexpected ERD typeCode %" PRId64 " while reading XML data";
+ case ERR_XML_GONE:
+ return "ran out of XML data";
+ case ERR_XML_INPUT:
+ return "unable to read XML data from input file";
+ case ERR_XML_LEFT:
+ return "did not consume all of the XML data, '%s' left";
+ case ERR_XML_MISMATCH:
+ return "found mismatch between XML data and infoset '%s'";
+ case ERR_XML_WRITE:
+ return "error writing XML document";
+ default:
+ assert("invalid code" && 0);
+ return "unrecognized error code, shouldn't happen";
+ }
+}
+
+// print_maybe_stop - print a message and maybe stop the program
+
+static void
+print_maybe_stop(const Error *err, int status)
+{
+ const int errnum = 0;
+ const char *format = "%s";
+ const char *msg = error_message(err->code);
+
+ switch (err->code)
+ {
+ case ERR_FILE_OPEN:
+ case ERR_FIXED_VALUE:
+ case ERR_INFOSET_READ:
+ case ERR_INFOSET_WRITE:
+ case ERR_STRTOBOOL:
+ case ERR_STRTOD_ERRNO:
+ case ERR_STRTOI_ERRNO:
+ case ERR_STRTONUM_EMPTY:
+ case ERR_STRTONUM_NOT:
+ case ERR_STRTONUM_RANGE:
+ case ERR_XML_ELEMENT:
+ case ERR_XML_LEFT:
+ case ERR_XML_MISMATCH:
+ error(status, errnum, msg, err->s);
+ break;
+ case ERR_CHOICE_KEY:
+ case ERR_PARSE_BOOL:
+ case ERR_XML_ERD:
+ error(status, errnum, msg, err->d64);
+ break;
+ default:
+ error(status, errnum, format, msg);
+ break;
+ }
+}
+
+// get_diagnostics - get pointer to validation diagnostics
+
+Diagnostics *
+get_diagnostics(void)
+{
+ static Diagnostics diagnostics;
+ return &diagnostics;
+}
+
+// add_diagnostic - add a new error to validation diagnostics
+
+bool
+add_diagnostic(Diagnostics *diagnostics, const Error *error)
+{
+ if (diagnostics && error)
+ {
+ if (diagnostics->length < LIMIT_DIAGNOSTICS)
Review comment:
Where there discussions about silently dropping diagnostics? What was
the decision for that?
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/Runtime2DataProcessor.scala
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.io.InputStream
+import java.io.OutputStream
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.DaffodilTunables
+import org.apache.daffodil.api.DataLocation
+import org.apache.daffodil.api.ValidationMode
+import org.apache.daffodil.externalvars.Binding
+import org.apache.daffodil.processors.Failure
+import org.apache.daffodil.processors.ProcessorResult
+import org.apache.daffodil.processors.Success
+import org.apache.daffodil.processors.VariableMap
+import org.apache.daffodil.processors.WithDiagnosticsImpl
+import org.apache.daffodil.processors.parsers.ParseError
+import org.apache.daffodil.processors.unparsers.UnparseError
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.Nope
+
+/**
+ * Effectively a scala proxy object that does its work via the underlying
C-code.
+ * Will need to consider how to use features of underlying C-code to get
infoset,
+ * walk infoset, generate XML for use by TDML tests.
+ */
+class Runtime2DataProcessor(executableFile: os.Path) extends
DFDL.DataProcessorBase {
Review comment:
Lot's of uncovered lines in this file. Is the purpose of this file to be
used when running with the "daffodil-runtime2" implementation in the TDML
runner? Should these all just output warnings that they are unsupported? I
*think* that's what we do in the IBM runner for example.
##########
File path:
daffodil-tdml-lib/src/main/scala/org/apache/daffodil/tdml/processor/TDMLDFDLProcessor.scala
##########
@@ -124,7 +124,10 @@ trait TDMLResult {
def isValidationError: Boolean
def isProcessingError: Boolean
def getDiagnostics: Seq[Diagnostic]
-
+ /**
+ * Deletes any temporary files that have been generated
+ */
+ def cleanUp(): Unit
Review comment:
Can we give a default implementation of nothing, e.g.:
```scala
def cleanUp(): Unit = {}
```
Otherwise, I *think* we'll break the IBM TDML Runner which doesn't have this
method defined. Or I guess we could just add that. Seems like we don't really
need to require a cleanup implementation if there's nothing to clean up though.
##########
File path: project/Dependencies.scala
##########
@@ -22,6 +22,7 @@ object Dependencies {
lazy val common = core ++ infoset ++ test
lazy val core = Seq(
+ "com.lihaoyi" %% "os-lib" % "0.7.1", // for generating C source files
Review comment:
Latest version of 0.7.3. Might as well update now instead of waiting for
this to merge then getting a PR from scala steward.
##########
File path: daffodil-runtime2/src/main/resources/c/Makefile
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Here's how to compile the C sources into a program for running
+# parse/unparse tests (.dat <-> .xml)
+
+PROGRAM = ./daffodil
+HEADERS = libcli/*.h libruntime/*.h
+SOURCES = libcli/*.c libruntime/*.c
+INCLUDES = -I libcli -I libruntime
+CFLAGS = -g -Wall -Wextra
+LIBS = -lmxml
+
+$(PROGRAM): $(HEADERS) $(SOURCES)
+ $(CC) $(CFLAGS) $(INCLUDES) $(SOURCES) $(LIBS) -o $(PROGRAM)
+
+# Here's how to run parse/unparse tests (.dat <-> .xml, although you
+# will need to create the .dat and .xml files first)
+
+PARSE_DAT = parse.dat
+UNPARSE_XML = unparse.xml
+
+clean:
+ rm -f $(PROGRAM) test_$(PARSE_DAT) test_$(UNPARSE_XML)
+
+tests: parse-test unparse-test
+
+parse-test: $(PROGRAM)
+ $(PROGRAM) parse $(PARSE_DAT) -o test_$(UNPARSE_XML)
+ xmldiff $(UNPARSE_XML) test_$(UNPARSE_XML)
+
+unparse-test: $(PROGRAM)
+ $(PROGRAM) unparse $(UNPARSE_XML) -o test_$(PARSE_DAT)
+ diff $(PARSE_DAT) test_$(PARSE_DAT)
+
+# You will need the Mini-XML library and xmldiff - here's how to
+# install both in Ubuntu 20.04
+
+deps:
+ sudo apt install libmxml-dev xmldiff
Review comment:
Not sure this belongs in a makefile that should be operation system
agnostic. We have BUILD.md file that describes dependencies.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ElementBase.scala
##########
@@ -354,6 +354,38 @@ trait ElementBase
} else DataValue.NoValue
}
+ /**
+ * Is either DataValue[AnyRef] or DataValue.NoValue.
+ *
+ * The value will always be of the matching primitive types for the element,
and
+ * directly usable as the value of a simple-type element.
+ *
+ * When a value is used, it is created from the XSD fixed attribute of the
+ * element declaration, and that string cannot contain DFDL entities of any
kind,
+ * nor any PUA-remapped characters. This insures the fixed value can still be
+ * used for ordinary XML-schema validation outside of Daffodil/DFDL.
+ */
+ final lazy val fixedValue: DataValuePrimitiveNullable = {
+ if (hasFixedValue && (isScalar ||
isArrayWithAtLeastOneRequiredArrayElement)) {
Review comment:
I think this condition is wrong for fixed values? Looks like this was
copied from default values, but I think the logic is different. I think maybe
this should just be ``if (hasFixedValue && isSimpleType)``? Even if an array is
entirely optional, we still want to know if it has a fixed value, and all
elements of that array should be checks against that fixed value during
validation. I think all we care about is if it's a simple type? Might not even
care about that, since complex types can't even have fixed attributes?
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/daffodil_argp.c
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h"
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse,
ARGP_ERR_UNKNOWN, ARGP_IN_ORDER, ARGP_KEY_ARG, argp, argp_option, ARGP_KEY_END
+#include <stdio.h> // for sprintf, NULL
+#include <stdlib.h> // for putenv
+#include <string.h> // for strlen, strcmp
+
+// Initialize our "daffodil parse" CLI options
+
+struct daffodil_parse_cli daffodil_parse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option parse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0, "Infoset type to output. Must
be one of 'xml' or 'null'", 0},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to a given file. If not given or is -, output is written to
"
+ "stdout",
+ 0},
+
+ {0}};
+
+static error_t parse_handler(int key, char *arg, struct argp_state *state);
+
+static const char parse_args_doc[] = "[infile]";
+
+static const char parse_doc[] = "\n"
+ "Parse a file using a DFDL schema\n"
+ "\n"
+ "Parse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to
parse. "
+ "If not specified, or a value of -, reads from
stdin";
+
+static const struct argp parse_argp = {
Review comment:
Just curious since I'm not too fimilar with FPGA's. Is this CLI
primiarily used to simplify testing and user friendliess, and an FPGA would
have a different entrypoint in the program (if so, where?), or in the FPGA
usecase is it essentially just executing the CLI and providing the right args?
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/daffodil_argp.c
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h"
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse,
ARGP_ERR_UNKNOWN, ARGP_IN_ORDER, ARGP_KEY_ARG, argp, argp_option, ARGP_KEY_END
Review comment:
Mentioned elsehwere, but I have concerns about this argp dependency. Is
there an Apache licensed dep we could use instead? Maybe we can get away with
getopt?
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/xml_writer.c
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_writer.h"
+#include <assert.h> // for assert
+#include <mxml.h> // for mxmlNewOpaquef, mxml_node_t, mxmlElementSetAttr,
mxmlGetOpaque, mxmlNewElement, mxmlDelete, mxmlGetElement, mxmlNewXML,
mxmlSaveFile, MXML_NO_CALLBACK
+#include <stdbool.h> // for bool
+#include <stdint.h> // for int16_t, int32_t, int64_t, int8_t, uint16_t,
uint32_t, uint64_t, uint8_t
+#include <stdio.h> // for NULL
+#include <string.h> // for strcmp
+#include "errors.h" // for Error, ERR_XML_DECL, ERR_XML_ELEMENT,
ERR_XML_WRITE, LIMIT_XML_NESTING, Error::(anonymous)
+#include "stack.h" // for stack_is_empty, stack_pop, stack_push, stack_top,
stack_init
+
+// Push new XML document on stack (note the stack is stored in a
+// static array which could overflow and stop the program; it also
+// means none of those functions are thread-safe)
+
+static const Error *
+xmlStartDocument(XMLWriter *writer)
+{
+ static mxml_node_t *array[LIMIT_XML_NESTING];
+ stack_init(&writer->stack, array, LIMIT_XML_NESTING);
+
+ mxml_node_t *xml = mxmlNewXML("1.0");
+ if (xml)
+ {
+ stack_push(&writer->stack, xml);
+ return NULL;
+ }
+ else
+ {
+ static Error error = {ERR_XML_DECL, {NULL}};
+ return &error;
+ }
+}
+
+// Pop completed XML document off stack and write it to stream (note
+// stack underflow will stop program)
+
+static const Error *
+xmlEndDocument(XMLWriter *writer)
+{
+ mxml_node_t *xml = stack_pop(&writer->stack);
+ assert(stack_is_empty(&writer->stack));
+
+ int status = mxmlSaveFile(xml, writer->stream, MXML_NO_CALLBACK);
+ if (status < 0)
+ {
+ static Error error = {ERR_XML_WRITE, {NULL}};
+ return &error;
+ }
+ mxmlDelete(xml);
+ return NULL;
+}
+
+// Push new complex element on stack (note stack overflow will stop
+// program)
+
+static const Error *
+xmlStartComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *parent = stack_top(&writer->stack);
+ const char * name = get_erd_name(base->erd);
+ const char * xmlns = get_erd_xmlns(base->erd);
+ mxml_node_t *complex = mxmlNewElement(parent, name);
+ if (xmlns)
+ {
+ const char *ns = get_erd_ns(base->erd);
+ mxmlElementSetAttr(complex, xmlns, ns);
+ }
+ stack_push(&writer->stack, complex);
+ return NULL;
+}
+
+// Pop completed complex element off stack (note stack underflow will
+// stop program)
+
+static const Error *
+xmlEndComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *complex = stack_pop(&writer->stack);
+
+ const char *name_from_xml = mxmlGetElement(complex);
+ const char *name_from_erd = get_erd_name(base->erd);
+ assert(strcmp(name_from_xml, name_from_erd) == 0);
+
+ return NULL;
+}
+
+// Fix a real number to conform to xsd:float syntax if needed
+
+static void
+fixNumberIfNeeded(const char *text)
+{
+ if (text[0] == 'N' && text[1] == 'A')
+ {
+ // xsd:float requires NaN to be capitalized correctly
+ char *modifyInPlace = (char *)text;
+ modifyInPlace[1] = 'a';
+ }
+ // These are not required by xsd:float, only to match runtime1 better
+ // - Strip + from <f>E+<e> to get <f>E<e>
+ // - Add .0 to 1 to get 1.0
+ // It would be better to compare floats as numbers, not strings, though
Review comment:
Was the code removed that this comment refers to?
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
Review comment:
The scala.util.Properties object has as isWin function. Basically the
same thing, but less we have to maintain if something with os.name ever
changes, assume scala keeps up with it.
##########
File path: daffodil-runtime2/src/main/resources/c/libruntime/errors.c
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "errors.h"
+#include <assert.h> // for assert
+#include <error.h> // for error
+#include <inttypes.h> // for PRId64
+#include <stdbool.h> // for bool, false, true
+#include <stdio.h> // for NULL, feof, ferror, FILE, size_t
+#include <stdlib.h> // for EXIT_FAILURE
+
+// error_message - return an internationalized error message
+
+static const char *
+error_message(enum ErrorCode code)
+{
+ switch (code)
+ {
+ case ERR_CHOICE_KEY:
+ return "no match between choice dispatch key %" PRId64 " and any
branch key";
+ case ERR_FILE_CLOSE:
+ return "error closing file";
+ case ERR_FILE_FLUSH:
+ return "error flushing stream to file";
+ case ERR_FILE_OPEN:
+ return "error opening file '%s'";
+ case ERR_FIXED_VALUE:
+ return "value of element '%s' does not match value of its "
+ "'fixed' attribute";
+ case ERR_INFOSET_READ:
+ return "cannot read infoset type '%s'";
+ case ERR_INFOSET_WRITE:
+ return "cannot write infoset type '%s'";
+ case ERR_PARSE_BOOL:
+ return "error parsing binary value %" PRId64 " as either true or
false";
+ case ERR_STACK_EMPTY:
+ return "stack empty, stopping program";
+ case ERR_STACK_OVERFLOW:
+ return "stack overflow, stopping program";
+ case ERR_STACK_UNDERFLOW:
+ return "stack underflow, stopping program";
+ case ERR_STREAM_EOF:
+ return "EOF in stream, stopping program";
+ case ERR_STREAM_ERROR:
+ return "error in stream, stopping program";
+ case ERR_STRTOBOOL:
+ return "error converting XML data '%s' to boolean";
+ case ERR_STRTOD_ERRNO:
+ return "error converting XML data '%s' to number";
+ case ERR_STRTOI_ERRNO:
+ return "error converting XML data '%s' to integer";
+ case ERR_STRTONUM_EMPTY:
+ return "found no number in XML data '%s'";
+ case ERR_STRTONUM_NOT:
+ return "found non-number characters in XML data '%s'";
+ case ERR_STRTONUM_RANGE:
+ return "number in XML data '%s' out of range";
+ case ERR_XML_DECL:
+ return "error making new XML declaration";
+ case ERR_XML_ELEMENT:
+ return "error making new XML element '%s'";
+ case ERR_XML_ERD:
+ return "unexpected ERD typeCode %" PRId64 " while reading XML data";
+ case ERR_XML_GONE:
+ return "ran out of XML data";
+ case ERR_XML_INPUT:
+ return "unable to read XML data from input file";
+ case ERR_XML_LEFT:
+ return "did not consume all of the XML data, '%s' left";
+ case ERR_XML_MISMATCH:
+ return "found mismatch between XML data and infoset '%s'";
+ case ERR_XML_WRITE:
+ return "error writing XML document";
+ default:
+ assert("invalid code" && 0);
+ return "unrecognized error code, shouldn't happen";
+ }
+}
+
+// print_maybe_stop - print a message and maybe stop the program
+
+static void
+print_maybe_stop(const Error *err, int status)
+{
+ const int errnum = 0;
+ const char *format = "%s";
+ const char *msg = error_message(err->code);
+
+ switch (err->code)
+ {
+ case ERR_FILE_OPEN:
+ case ERR_FIXED_VALUE:
+ case ERR_INFOSET_READ:
+ case ERR_INFOSET_WRITE:
+ case ERR_STRTOBOOL:
+ case ERR_STRTOD_ERRNO:
+ case ERR_STRTOI_ERRNO:
+ case ERR_STRTONUM_EMPTY:
+ case ERR_STRTONUM_NOT:
+ case ERR_STRTONUM_RANGE:
+ case ERR_XML_ELEMENT:
+ case ERR_XML_LEFT:
+ case ERR_XML_MISMATCH:
+ error(status, errnum, msg, err->s);
+ break;
+ case ERR_CHOICE_KEY:
+ case ERR_PARSE_BOOL:
+ case ERR_XML_ERD:
+ error(status, errnum, msg, err->d64);
+ break;
+ default:
+ error(status, errnum, format, msg);
+ break;
+ }
+}
+
+// get_diagnostics - get pointer to validation diagnostics
+
+Diagnostics *
+get_diagnostics(void)
+{
+ static Diagnostics diagnostics;
+ return &diagnostics;
+}
+
+// add_diagnostic - add a new error to validation diagnostics
+
+bool
+add_diagnostic(Diagnostics *diagnostics, const Error *error)
+{
+ if (diagnostics && error)
+ {
+ if (diagnostics->length < LIMIT_DIAGNOSTICS)
+ {
+ Error *err = &diagnostics->array[diagnostics->length++];
+ err->code = error->code;
+ err->s = error->s;
+ return true;
+ }
+ }
+ return false;
+}
+
+// print_diagnostics - print any validation diagnostics
+
+void
+print_diagnostics(const Diagnostics *diagnostics)
+{
+ if (diagnostics)
+ {
+ for (size_t i = 0; i < diagnostics->length; i++)
+ {
+ const Error *error = &diagnostics->array[i];
+ print_maybe_stop(error, 0);
+ }
+ }
+}
+
+// continue_or_exit - print and exit if an error occurred or continue otherwise
+
+void
+continue_or_exit(const Error *error)
+{
+ if (error)
Review comment:
Back when I did C, I was more used to seeing code like
```c
if (!error) return;
print_maybe_stop(error, EXIT_FAILURE);
```
Is this an FPGA thing, stylistic thing, or something else?
##########
File path:
daffodil-tdml-lib/src/main/scala/org/apache/daffodil/tdml/TDMLRunner.scala
##########
@@ -477,7 +476,13 @@ abstract class TestCase(testCaseXML: NodeSeq, val parent:
DFDLTestSuite)
lazy val tdmlDFDLProcessorFactory: AbstractTDMLDFDLProcessorFactory = {
import scala.language.existentials
- val className =
"org.apache.daffodil.tdml.processor.TDMLDFDLProcessorFactory"
+ // tdmlImplementation is a tunable choice with three values.
+ val className = tunableObj.tdmlImplementation match {
+ // Right now daffodil and ibm use the same ProcessFactory name
+ case "daffodil" | "ibm" =>
"org.apache.daffodil.tdml.processor.TDMLDFDLProcessorFactory"
+ case "daffodil-runtime2" =>
"org.apache.daffodil.tdml.processor.Runtime2TDMLDFDLProcessorFactory"
+ case other => Assert.invariantFailed("'%s' not valid for
tdmlImplementation".format(other))
Review comment:
Ah, so I guess this is why we need to include the different imlementions
in a the tdml.xsd file. Because these class names are hardcoded rather than
using something like SPI to find the right class.
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
+ // Used by WithDiagnostics methods
+ private var diagnostics: Seq[Diagnostic] = Nil
+ private var errorStatus: Boolean = false
Review comment:
These vars make this so this isn't thread safe. I'm not sure that's a
big deal since I imagine most most people aren't going to use this in a thread
environment, and it's not even part of the public API so perhaps this is
considered use at your own risk, but maybe something to consider, or at least
document somewhere. I gues we just need to make sure where we do code
generation, we also create a new code generator, since we might not know if
it's being run in a thread or not?
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/Runtime2CodeGenerator.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import org.apache.daffodil.grammar.Gram
+import org.apache.daffodil.grammar.Prod
+import org.apache.daffodil.grammar.RootGrammarMixin
+import org.apache.daffodil.grammar.SeqComp
+import org.apache.daffodil.grammar.primitives.BinaryBoolean
+import org.apache.daffodil.grammar.primitives.BinaryDouble
+import org.apache.daffodil.grammar.primitives.BinaryFloat
+import org.apache.daffodil.grammar.primitives.BinaryIntegerKnownLength
+import org.apache.daffodil.grammar.primitives.CaptureContentLengthEnd
+import org.apache.daffodil.grammar.primitives.CaptureContentLengthStart
+import org.apache.daffodil.grammar.primitives.CaptureValueLengthEnd
+import org.apache.daffodil.grammar.primitives.CaptureValueLengthStart
+import org.apache.daffodil.grammar.primitives.ElementCombinator
+import org.apache.daffodil.grammar.primitives.ElementParseAndUnspecifiedLength
+import org.apache.daffodil.grammar.primitives.OrderedSequence
+import org.apache.daffodil.grammar.primitives.ScalarOrderedSequenceChild
+import org.apache.daffodil.grammar.primitives.SpecifiedLengthImplicit
+import org.apache.daffodil.runtime2.generators.BinaryBooleanCodeGenerator
+import org.apache.daffodil.runtime2.generators.BinaryFloatCodeGenerator
+import
org.apache.daffodil.runtime2.generators.BinaryIntegerKnownLengthCodeGenerator
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import
org.apache.daffodil.runtime2.generators.ElementParseAndUnspecifiedLengthCodeGenerator
+import org.apache.daffodil.runtime2.generators.OrderedSequenceCodeGenerator
+import org.apache.daffodil.runtime2.generators.SeqCompCodeGenerator
+import org.apache.daffodil.util.Misc
+
+import scala.annotation.tailrec
+
+object Runtime2CodeGenerator
+ extends BinaryBooleanCodeGenerator
+ with BinaryIntegerKnownLengthCodeGenerator
+ with BinaryFloatCodeGenerator
+ with ElementParseAndUnspecifiedLengthCodeGenerator
+ with OrderedSequenceCodeGenerator
+ with SeqCompCodeGenerator {
+
+ @tailrec
+ def generateCode(gram: Gram, state: CodeGeneratorState): Unit = {
+ gram match {
+ case g: RootGrammarMixin =>
Runtime2CodeGenerator.generateCode(g.documentElement, state)
+ case g: Prod if (g.guard) => Runtime2CodeGenerator.generateCode(g.gram,
state)
+ case g: ElementCombinator =>
Runtime2CodeGenerator.generateCode(g.subComb, state)
+ case g: SpecifiedLengthImplicit =>
Runtime2CodeGenerator.generateCode(g.eGram, state)
+ case g: ScalarOrderedSequenceChild =>
Runtime2CodeGenerator.generateCode(g.term.termContentBody, state)
+ case g: BinaryBoolean => binaryBooleanGenerateCode(g.e, state)
+ case g: BinaryDouble => binaryFloatGenerateCode(g.e, 64, state)
+ case g: BinaryFloat => binaryFloatGenerateCode(g.e, 32, state)
+ case g: BinaryIntegerKnownLength =>
binaryIntegerKnownLengthGenerateCode(g, state)
+ case g: ElementParseAndUnspecifiedLength =>
elementParseAndUnspecifiedLengthGenerateCode(g, state)
+ case g: OrderedSequence => orderedSequenceGenerateCode(g, state)
+ case g: SeqComp => seqCompGenerateCode(g, state)
+ case _: CaptureContentLengthStart => noop
+ case _: CaptureContentLengthEnd => noop
Review comment:
I'm surprised some of the above cases aren't ever hit. Is it not
possible or are tests needed for these cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
